Don’t get me wrong – you can get away with running a site on almost zero dollars, and we’ve all heard about those sites running on someone’s PC under their desk… but what I’m touching on here is a ‘real’ website.

I’m the only developer on a website which is somewhat large. Its no where near the level of the big news sites, or the social sites etc, but we get a fair amount of traffic and the codebase is big enough to keep me busy all the time. And to be honest my life would be much easier if I could just push out the latest version of the site and ignore everything else.

But lets be honest – large websites are complicated, there is always the change (more often then not) that there might be some bugs laying around, and on top of this performance can be an issue. And don’t even get me started on email and advertising. So what I now hope to accomplish is an inventory of all the ‘extras’ which seem to get over looked, but are quite important, for a production website.

Exception Handling

No one likes to see error messages, but they’re unfortunately one of those things we have to live with – lets be honest, no one is perfect (I’m certainly not) – but even though we cant guarantee that errors won’t be a problem we should be ready to handle them properly and log them, so when the time comes and you have angry users complaining that your website isn’t working you can be rest-assured that you have any error details available to you without the hassle of asking the user what the error message was, what they were doing, what color t-shirt they had on… you get the picture.

There are a couple of options here, but the one I use for everything is HopToad from Thoughtbot – this is an extremely simple tool to use. Once your signed up online they give you some quick instructions which includes installing a plugin into your Rails application, then running a generator with an api key for your project, and then thats it – once you go live and if an error occurs Hoptoad sends you a nice email and makes all the details, including a backtrace, the URL the error happened at, the referring page, the environment variables etc, online and within reach.

There are, of course, free alternatives around, and if you really wanted you could just have your app email you when an error occurs, but I’m sure most people can relate to the situation where your inbox becomes flooded because you forgot to check for a nil? Yeah, Hoptoad prevents this exact problem.

And the cost? $15/month, for 16 projects – very affordable for what it provides, and there’s some extra bonuses you’ll stumble upon which make you think ‘wow this is worth it’.

Application Performance

So, errors we’ve covered. Next question – do you know how well your website is running? How do you know which areas are the slowest to run, and do you know what the user experience is like?

On this front as well there are a few options – you could quite easily run Request Log Analyzer and pass it your latest production log file, but thats not really a real time view, and if you want to look through historical periods its somewhat difficult, involving splitting out your logs based on time stamps – not my idea of fun.

RPM to the rescue – NewRelic RPM to be exact. I suppose that this is probably the best known ‘Rails’ application around. Mind you, its not cheap, but for what it provides its actually invaluable for popular websites.  And what does it provide? Real time metrics, drill down trouble shooting, pretty graphs and more. Really, I’d just recommend that you give their free version a go – I’d wager that you will want to upgrade to get the extra features once you get a taste.

And on top of providing a great product they also sponsored the Scaling Rails screen cast series with Gregg Pollack – I’d highly recommend everyone checks out this series.

So how much will this set you back? It varies based on pricing level, but currently we pay about $200/month – and its worth every cent.

Server Monitoring

Now we have errors covered off and application performance is taken care of what about your hosting infrastructure? If your using a Rails hosting specialist like Engine Yard or Rails Machine or similar then they’ve got this covered, skip on to the next point, but if you’ve rolled your own, be it with your own hardware, or on cloud infrastructure like EC2, it would be very wize to start collecting some metrics about how your machines are actually running.

Of course, there is a myriad of options here, so let me just focus on what we use, which is Server Density – these guys make monitoring, alerting and metric collection as simple as making a sandwich (well, as long as it isn’t some fancy sandwich which requires some weird condiment you can only get by milking deep sea predator fish…). Sign up for an account, setup a server in their system, and once you’ve installed the Server Density agent, drop in the config details and wait.

Its extremely easy to setup alerts based on thresholds (such as memory free, CPU load, number of request for nginx or apache, even connections to MySQL or MongoDB), and they support notifications via Email, SMS and iPhone push as well (though to be honest, if you publish something controversial which generates some traffic the alerts can become a bit irritating). And if you want to monitor something they don’t support, and your a bit of a dab hand at Python, they have a brand spanking new plugin framework for you to use.

This has saved me a few times, especially being able to look back in time, see a spike, check out what was happening on my system at that time, and the simplicity is just awesome. Plus, they’ve been public adopters of MongoDB for their data collection and feedback to the community with their experiences.

Server Density – $10 (per server), being able to tell your boss that the servers aren’t really on fire… priceless (yeah thats kind of lame, sorry).

Email

I’ve unfortunately discovered, more often then not, that sending emails from web applications is a bit of a thorn in the side (sure, Rails makes this a lot easier with ActionMailer providing views etc). Its not actually the sending thats the issue – anyone can put together some content and use a library like tmail, Pony, even ActionMailer to shoot off a message – the real issue is dealing with delivery problems.

Running on Amazon EC2, while amazingly flexible and cost effective, has a slight issue – the same things that make EC2 appealing to me also make EC2 appealing to spammers. And as a result there are a lot of RBLs, block lists etc, which don’t like emails from the EC2 address space. Plus, PTR records, you have no control over them.

So why risk the emails not being sent? And what about bounces – soft and hard, plus do you know what happened with the email after it was received?

I was initially introduced to SendGrid through Heroku (that is another topic all together) – before that I was hacking about with smtp-tls and Google Apps SMTP to get my emails out. For the site that I work on now there were two options – use the SMTP server provided by the hosting company (before we moved to EC2), setup our own MTA, or use an external service. The decision time there was less then a second – straight away I was on the SendGrid page and registering for the company.

SendGrid makes things simple – I tell my app which SMTP settings to use, which authenticate, of course, and then I get everything I need from SendGrid straight away – I can turn on open and click tracking, so I know if users are ‘doing things’ with the emails we send out, we have domains keys enabled automatically, plus we also get to see if there are any bounces of emails, and compare that to our outgoing amount (more shiny graphs).

And the total cost for this is $9 per month - perfect.

To summarize this hasn’t covered everything we pay for externally – there are tools like Basecamp (for the higher level project management and collaboration), Unfuddle (for the development project management), Zendesk (for helpdesk.. those pesky users) and MailChimp (for email campaigns) we use, but its a case of ‘different strokes for different folks’ – but what I’ve mentioned above are just the basics.

The real take away message here is that there are always other things you need to factor in when your considering when you figure out how much running a website is actually going to cost you. And if your wondering why you don’t just build this stuff your self.. well, sure, I’m not stopping you, I just feel that most times its smarter to pay for something already available rather then spending time reinventing the wheel, only to realize what you finished up with is missing a whole lot of stuff.

Unfortunately, the current attempt is very poorly done. I’m not blaming the developers, in fact I can quite easily see how this could be an afterthought or something which wasn’t high up on the priorities for this project at the moment (after all, launching the new site is the first major step, providing content to mobile devices can come later…) – but, the good news is that there is a simple and quick fix…

Introducing… jQTouch – its really as simple as writing some HTML (and, perhaps sometimes some Javascript…).

Ok, so if you were nodding your head to the above question, do you craft up your email in your desktop or web-based email editor, put all the recipient addresses in the BCC field and hit send?

‘What!?’ you say, ‘You shouldn’t be charging to support IE6, it should just be included!’

Well, yes, it should be included – if this were still 2005. But, its now 2009, and Internet Explorer 6 is over 8 years old (released in 2001), and its also due to finish its support lifetime with Microsoft next year – but its not just because it is old.

Internet Explorer 6 is a huge pain in the proverbial. Granted, things have gotten much better as IE7 and now IE8 have been released, however the problem still exists with the fact that a large number of government departments are still 100% reliant on IE6 and are trying to avoid upgrading at any cost.

I could rant on about this for hours, but its probably best to put this into more of a fiscal perspective – where, exactly, does the fee for IE6-compatible sites go?

The normal process of taking a website from design as a Photoshop design and transforming it into HTML templates usually touches on the following steps:

• Review of the PSD file, and, if required, labeling and grouping layers to streamline the slicing process.
• Performing layer slicing and element slicing, naming the slices (the ‘cut up’ sections of the design) and then exporting as PNG files, or JPEG files if they are photographs.
• Coding up the basic structure of the page, and the base stylesheets. If the design was done using a design-grid in mind this process can be sped up by using a CSS framework (basically a pre-rolled set of stylesheets for layout) such as Blueprint, 960 Grid System or otherwise.
• Layout of image element, whether they are background images, positional images, navigation buttons etc.
• Adjusting the text look & feel (the typography), tweaking things like line spacing, letter spacing, font weights and margins between sections.
• Final polish and testing, including browser tests across platforms. Usually for this I will use something like Litmus which allows me to test a page over 20+ different browsers – or, if required I can test manually using virtual machines.

Usually, this is all I need to do. The end result is then handed off to the client after they sign off, and the world is at peace. Unfortunately this isn’t all if IE6 support is a requirement.

First off the bat is IE6’s abysmal support for transparent PNGs. Back in 2001 PNG was still an obscure image format being developed to overcome licensing limitations of GIF. While it wasn’t unused, it wasn’t as popular as GIF, and therefor IE6 didn’t have full support for PNGs features, such as full 8-bit transparency. The result? Viewing a PNG in IE6 would show up with a nasty grey ‘thingie’ wherever transparency was supposed to show up.

There is, however, a solution for this – there is a DirectX filter which can be loaded for the image which will apply the transparency. The problem is that it is a HUGE hack, and this has to be done for every PNG file loaded. The advent of JavaScript libraries such as PngFix or jQuery plugins to do the same thing help out here, but its still a bit dodgy and just doesn’t feel right.

The alternative here is to use GIFs instead of PNG, however GIF only supports 1 level of transparency (transparent, or opaque) which doesn’t give a whole lot of flexibility to the demands of modern web design.

Another issue is the support for CSS standards in Internet Explorer 6. There are numerous hacks within CSS to workaround, fix or trick IE into working with certain styles. Now, I can’t claim that IE is the only wrong-doer here – back in the ‘day’ it seemed that it was each to his own, but browsers are coming into line and ‘playing’ nice with CSS standards now, and considering that the last service pack for IE6 was released in 2008 you would think that maybe they could include somehow the ability to signal to the browser that we want them to render stuff properly?

Alas, the work around is: Develop your stylesheet, make sure it looks right in everything other then IE6. Then, look at it in IE6, and spend hours figuring out how to apply styles, hacks and whatever to get around the layout and presentational issues which IE6 causes, which you would then need to put into a separate stylesheet only to be loaded if IE6 is present, otherwise these adjustments will alter the ‘working’ layout files.

One of the issues with this is that the tooling available for sorting out these issues is very lacking. Normally I would open up Firefox, load up Firebug and debug away, but in IE6 the most we have is the Internet Explorer Developer Toolbar, which, while more useful then nothing, is still like trying to climb a very slippery slope.

There is one terrible work around for a lot of positional issues like this – use tables. But now, everyone knows (or should know) that tables should only be used for Tabular data, not for layouts. Sorry, but its CSS all the way for me

The final thing I think I should touch on is JavaScript. Once again, everything is different, but coding to support IE, Mozilla, Opera, Safari etc etc was just hell some times. And certain DOM calls would only work in certain browsers. I still have memories of the browser-detection scripts to try and get some insight as to what is running to try and be as error-free as possible.

Thankfully now there is jQuery, which, I think I can honestly say, is the best JavaScript abstraction layer around today – if something like this had been around when I first started playing with Web Development I think everything would have been a lot less frustrating.

So, I guess now you can probably see why the extra charge.. But maybe you are wondering why not just include it as extra hours? Well, the thing is that I feel, as a web developer, it is partially my responsibility to educate my clients about these issues – continuing to use IE6 is hindering the web, development and growth. It encourages bad practices and it enables Government departments to be lazy with their web application development. If I could see a time where I never have to see a notice like ‘this website works best in Internet Explorer 6 or Netscape Navigator’ it will be a very happy day indeed (yes, ATO, I am looking at you and the ABR portal).

So, putting a charge on my quotes which says ‘Internet Explorer 6 Support/Compatibility’ is my way of raising awareness.

This actually turned into a bit of a rant, but I guess it couldn’t hurt to say my piece…

• Hoptoad
• Google Analytics
• RPM

However when you’re breaking into a new market its a bit hard to get referrals, word of mouth advertising, or even testimonials if you haven’t work for anyone in that market – so what do you do?

I’ve ended up in this situation – the Swellnet website was a project I inherited and is running a combination of Prototype and jQuery, and the scripts are all over the place. So, tonights project, before I start making the changes I have queued up, will be to try and untangle this mess.

So lets dig into it – but some points to note first:

• I’m not unit testing – if there were JS unit tests setup originally I would, of course, continue using them, however to create them at this point would be a futile effort, and I’d much rather go back and do this after the clean up.
• Number of files, and file sizes aren’t necessarily a concern – we use asset_packager to bundle and pack our files into defined singular files (e.g. base libs etc).
• Your means might vary – what works for me may not make sense to you, or just feel awkward – this is just a suggestion and showing you how I’ve accomplished the task.

Ok.

Remove duplication

First step is lets get rid of the extra framework. Personally I dislike Prototype – I looked at it a few times, but it always seemed much more verbose then jQuery, and the whole loading up 5 different files for effects etc? Sorry, not my idea of fun.

Currently we are running jQuery in compatibility mode, which means that all calls to jQuery use the jQuery function name, instead of the $ alias. This is helpful because we can look for calls out to Prototype by identifying the $ calls. (Keep in mind that jQuery plugins etc will reference $ internally through the anonymous function in the plugin pattern).

Files Removed: prototype.js, scriptaculous.js, controls.js, effects.js, sound.js, dragdrop.js, slider.js, builder.js, carousel.js

Pulling out these files now means that there are a bunch of things dependent on the $ alias which now don’t work – good! If this wasn’t the case there’s something weird going on

So, first off one of the plugins we use for tabs is failing to run. The method used to initialize the tabs uses the dom:loaded event, which we replace with jQuery(function(){ ... }) – however thats not all we need to fix.

Fixing up plugins

The prototype plugins I’ve come across in this project are more or less classes, with their own functions in the Class.prototype object, and while its entirely possible to mush this into working with jQuery the best option is to rewrite it into a jQuery plugin.

The code before:

The code after:

Not a huge change, but the main thing is that the initialization code, which is at the footer of the first Gist, has now become: jQuery("#newstabs, #hometabs, #searchtabs, #livebuoytabs").fabtabs(); – and it lives in the application.js file. Much nicer

The loaded events and scattered methods

I have a feeling that this is something I picked up a while ago which formed a bad habit. I’ve always hated the idea of the body onload="..." way of doing Javascript when the dom has loaded, and when jQuery came along and I was introduced to using jQuery(function() { ... }); I was glad to get away from the body tag, or attaching event handlers at the top or bottom of files etc.

But, unfortunately once I started using this everything went into that anonymous function. Sure, most of the stuff was jQuery operations, but when you have 20+ calls to jQuery, each with their own set of arguments, and most of them being anonymous functions, well… it gets a bit messy, and very hard to find what you’re looking for.

So, enter the hash.

From messy to somewhat organized.

I originally stumbled across this while watching a screencast from Peepcode on jQuery+AJAX (by the way, I would highly recommend a yearly subscriptions – completely worth it!). The idea is simple: create a hash object, and make the values anonymous functions:

There’s a lot of room for me to refactor this a lot more, especially with the anonymous functions for jQuery operations – but this is a good start. The main benefit is I can see all the initializers without the mess of the operational code, and its very easy to jump between calls in RubyMine (caret over the function name, Command+B, and you’re there).

The overall result is nicer code, easier to maintain and navigate, and I no longer try to avoid changing the javascript for this site (well, for the public section anyways, the admin side still needs to be looked at). And, on top of that, there was a size reduction of about 30k – not really a lot, but every little bit counts!

So, once I clean up the admin side of things (part of a bigger project) its onto refactoring our nasty nasty CSS into SASS – who wants to help? Its only ~10k lines (yes, I did say ten thousand lines… don’t ask…).

I’ve touch on business tools before, about how you really should have dedicated infrastructure like internet services and phone lines for your business, and not share this infrastructure with the rest of your home when you are in a home office (of course, certain things like electricity and water and heating etc are a bit hard to separate).

Today I’m going to cover a different angle – lets discuss the software you use on your business computer – and particularly – did you pay for it?

A question was posed to me by Joe about Microsoft Office, does it come with the laptop etc, which I answered explaining the licensing etc, and said person went out and bought a copy of office – all things are good in the world.

Then there was a need for Photoshop to be installed on the laptop – after all, the editing of graphics, photos etc are a core part of this persons business. This turned out to be quite an issue, because… well Photoshop is an expensive piece of software – the last time I checked it was retailing for about AU$1,200 – but hey, its a big important tool and I’m pretty sure it cost a lot of money to develop. However, Joe didn’t want to pay so much for this software, and I was asked if I had a copy, or where he could find it and so on.

Lets just take a step back for a minute and consider something – I am a software developer – my lively hood more or less relies on people paying for and using my software, either directly or indirectly – so if I were to assist Joe in obtaining this software without paying for it then I would pretty much be hurting other developers – do you believe in Karma?

So, as you might expect I told Joe he should purchase the software, which Joe didn’t like, and an argument started boiling up – and currently I don’t actually think a good outcome has been found (but that is neither here nor there…).

The point which I want to get across is: Sure, the software is expensive, but you use this as a core part of your business why shouldn’t you pay for it?

Lets say Joe has a client paying him $6000 for a job, and 50% of that job involves graphics work with Photoshop – theoretically you could say that Joe is making $3000 by using Photoshop for the clients work (well, and the labour involved), and that if he didn’t use Photoshop, or any other tool, he wouldn’t really get that $3000. In this case, why should Adobe be expected to not receive payment for the product they created which is helping Joe make money?

Business is all about doing things properly – the right way, and keeping things legit. We’ve all seen companies go down because they tried to take dodgy shortcuts, done poor work, failed to look after their staff or clients – so if you are running a business then you should probably do it the right way, and pay for the tools you use, right?

Unfortunately the fallacy is that computers have become cheap – this may be right for home use, but for business they are still a very important tool and they still cost money.

So, my message to take home here is if you are using a software product for your business, then you should pay for it and be properly licensed- how would you feel, or how much would your business suffer, if people were taking your creations without paying you and then using them to make money for themselves? Eventually what goes around will come back around.

Take note of this Joe, seriously, just pay for the software – you wont actually miss the $1,200, and its actually a business expense, so use it for taxation purposes.

Because I’m still not entirely comfortable with Objective-C, and I’m still reading up on the Cocoa framework I decided to build my app using MacRuby so I could blend what I’m good at with some new things as well, and a shiny interface.

The whole process was actually really good – XCode is set up with the MacRuby application templates when you install the latest build, and setting up IBOutlets and IBActions is as simple as using attr_accessor utlet_name and def action_name(sender) – how much simpler can you get?

So things were running pretty smoothly with my test commands, everything was returning quickly, the console output was being appended as expected… until I switched my commands to something which was a bit more longer running –

git push

. Why is this an issue? It all comes down to Blocking, and UI Responsiveness….

Unfortunately the backtick, popen and popen3 methods all hold onto the executing thread until they are done, and if this is in your main thread (the UI Thread), which would normally be the case if you are responding to a button click event, you going to see the ‘spinning beach ball of doom!’.

Fortunately in the latest release of MacRuby (currently 0.5 beta) support for Grand Central Dispatch (note: you’ll need Snow Leopard for this…) is included and makes the whole threading, synchronisation, locks etc as simple as a few lines of code… So how is it done?

    DISPATCH_QUEUE = Dispatch::Queue.new("net.amasses.app_name.queue_name")

    DISPATCH_QUEUE.dispatch do
        IO.popen(full_cmd) do |streamer|
          while line = streamer.gets
              Dispatch::Queue.main.dispatch { console_output.insertText line }
          end
        end

        Dispatch::Queue.main.dispatch do
          console_output.insertText "\r\r Done \r\r"
          go_button.setEnabled true
        end
    end

So, a few small points.
1. The DISPATCH_QUEUE.dispatch do block basically tells the runtime to execute anything within this block on the new queue thread. Yes, it is that simple.
2. If you want to do things with your UI you will need to switch back to the main (or UI) queue – this is done with Dispatch::Queue.main and it acts just like a regular dispatch queue.

Keep in mind that your means might vary, and I haven’t even dipped my toes into locks and synchronization with Semaphores, but from what I have seen (and how I understand it) it isn’t much harder then what you see here.

Enjoy!

MacRuby header image from http://blog.anthonyburns.co.uk/

Surprisingly enough, in 2009, this is still a very common occurrence – a virus or worm infection. If you’re running off one computer sometimes it isn’t so bad, or, sometimes (if you forgot to backup) it can be a catastrophe. What is even worse is if you are running on a network, which is almost like virus heaven – one machine becomes infected and suddenly it is spreading everywhere.

Unfortunately I become aware of an incident this morning affecting a small-medium company with about 3 networked servers and an unknown number of workstations, and they are, for the most part, infected with a virus/worm which is propagating via network shares, AND email, so, a large number of bouncing back emails of course. The fix? At the moment we aren’t sure – while the systems had Antivirus running, it obviously wasn’t good enough – whether that was because it was out of date or otherwise – and the decision will be made tomorrow if I should go and visit them to work on the system directly and, if need be, prepare for a rebuild of their systems.

Working previously as a computer technician, and later as a consultant, I was amazed at the number of people who didn’t use Antivirus products, or, who had a system which was out of date. Unfortunately the problem seems to be one of money. It seems that once $1500+ has been forked out for a system, along with software like office etc, the idea of having to pay another $100 for Antivirus tools, as well as pay a fee each year to download the updates automatically seems to be kicking sand in the face of the consumer.

Perhaps the issue here is education? When you buy a car you are required to get compulsory third party insurance before being allowed to register it – what if Antivirus was advertised as the same? What if, when you bought a computer, you paid an ‘insurance’ premium which included things like your antivirus subscriptions and whatever else is needed now days?

So, what should you really do?

Before anything at all I’d really really recommend backups. People might read this and start off on the ‘it is too hard’ excuse, but there are services which will backup all of your data, encrypted, off site for $5/month. Install it, set it up and forget – how hard can that be? And this kind of thing is really something you need before any potential problems occur – trust me on this one – there is no point backing up infected files…

The other thing is to keep your Antivirus tools up to date. Personally I don’t have a favourite at the moment – it has been a while since I’ve really been in the retail industry so I’m no longer qualified to give a full recommendation, but it isn’t hard to find what you need.

And if/when a virus strikes?

Call a professional – if your data is important to you I don’t think it is really an option to be worrying about finances. The longer you continue to use your system, the more time that the virus has to cause damage or make changes to your files.

It might help to think about it in the following terms: How much will 2 days of downtime cost you (or if you loose everything, how much will it cost to get back on board?) – then compare it to how much it would cost to have a professional work on your system for half a day at a minimum. I think you might find that hiring a professional will be much more affordable

As with everything, your means might vary, but its important to try and spend some time on this. I feel that people still see IT as an expense and something that doesn’t have a return, but now pretty much everything depends on IT you can’t afford to be without your systems…

One final thing I should probably mention is that 99.9% of the time when I deal with a virus infection I prefer to wipe the machine clean and start from scratch. While this seems somewhat drastic there is always a chance that reinfection will occur. So, something to keep in mind – you may be able to squash the infection but you never know when it might pop up again…