With the price of storage dropping all the time, there is a constant perception from people who don’t deal with it every day that “disk space is cheap”, especially when it comes to developers. The problem is that so called “Enterprise” storage costs are still astronomical compared to what people are used to paying for home storage – even when using SATA disks.
A lot of this extra cost comes from a perceived requirement for the highest available capacity, availability and performance. Achieving all three characteristics is expensive, but if you’re willing to sacrifice one of them then costs start to fall considerably. Lowering requirements on two of the three drops it even more.
One of the teams I work with has a requirement primarily on capacity. Performance and availability are nice, but capacity is the key. We generate gigabytes worth of log files every day, but didn’t have one place to store it all for easy analysis. Just before I joined the team they’d purchased the cheapest “Enterprise” storage system the IT team at the time would allow – it ended up costing in the region of £12k for 12TB of raw storage. That’s £1000 per TB!
In addition to the price, the other problems were accessibility and management of the data and managing growth. This inspired a hunt for something that would provide a cheaper and more flexible solution.
Our requirements were:
- *nix based system. The current storage solution was based on Windows Storage Server, but all our systems and tools for this team are Linux based. Yes, Windows does technically provide things like an NFS server, but fighting with the file system permissions and overall performance are two things that impacted us.
- Cheap to expand. We need to have a clear path to grow the storage in the server easily by simply adding more disks.
- Large filesystems. There’s nothing more wasteful from a storage point of view than having lots of small filesystems. Besides the management overhead, there’s also many wasted blocks lying around un-used.
- Cheap to build. This inevitably means commodity hardware.
- Reasonable availability. We don’t need 99.999% uptime, but would be happy with somewhere in the region of 90%+
- Reasonable performance. Primary access to the data on this machine is via gigabit Ethernet. As long as it can keep up with the network card we’re happy…
After more than 5 years at ThoughtWorks I’ve decided that it’s time for a change of scenery. As much as I enjoyed the challenge of consulting, meeting new people and seeing new places – I prefer spending time with my family more. It was a tough decision to make, but I think it’s the right one for me now. I will miss many people at TW, but on the plus side I’m again working with some great people who I missed when they left TW…
As I’m going to be doing DevOps type stuff all day every day at DRW now, I’m hoping I’ll have more time to document and share the things I discover.
Up until now I’ve managed to avoid ranting here, but I can no longer resist! After all these years of experience, why are the error messages in windows still generally meaningless? Is it because there are so many of them?
The problem I faced was trying to patch a game. Downloaded the patch, tried to apply it, but got the message “The application failed to initialize properly (0xc0000135)”. Much confusion. Downloaded from another source, tried again, same error. Tried on my laptop, and this time it worked perfectly. What’s the problem?
Turns out the patch is written in .Net, and I don’t have the correct version of the runtime installed on my desktop, but I do have it on my laptop. I only had .Net 1.0, and it needs a newer version. Good thing the error message told me that in the first place then, isn’t it…
A few of us are using Trac for an internal project at work, and at some stage in the process we decided to use the wiki as the distribution documentation. Everything has been going wonderfully, until we realized we that we wanted the documentation to be available without having to run Trac. After some digging around, I found this ticket on the Trac dev site where Alec Thomas put up a diff to allow you to download Trac wiki pages as HTML. After applying the patch, I found that it worked well, but there was still no easy way to pull the whole wiki down as HTML. So I wrote a python script to do it, and the script is available here for those who are interested.
So, after a short stab at blogging over at http://netnix.blogspot.com, I decided to start blogging at a domain name that was more me. Not because what I was writing about was of huge interest to people, or had lots to say. It’s just because I can. I’m going to leave the NetNix stuff where it is, as I don’t see the point of moving it. All new stuff will simply appear here.
Currently, the interesting stuff I’m working on is creating build production lines for software, and messing with virtualization. As soon as I have something interesting to report on either of these, you will see it here…