TechOpsGuys.com Diggin' technology every day

September 21, 2010

Online Schema Changes for MySQL

Filed under: General — Tags: , — Nate @ 9:08 pm

Looks like Facebook released a pretty cool tool that apparently provides the ability to perform MySQL schema changes online, something most real databases take for granted.

Another thing noted by our friends at The Register, was how extensively Facebook leverages MySQL. I was working on a project revolving around Apache Hadoop and someone that was involved with it was under the incorrect assumption that Facebook stores most of it’s data on Hadoop.

At Facebook, MySQL is the primary repository for user data, with InnoDB the accompanying storage engine.
[..]
All Callaghan will say is that the company runs “X thousands” of MySQL servers. “X” is such a large number, the company needed a way of making index changes on live machines.

I wouldn’t be surprised if they probably had a comparable number of MySQL servers to servers running Hadoop. After all Yahoo! is the biggest Hadoop user and at my last count had “only” about 25,000 servers running the software.

It certainly is unfortunate to see so many people out there see some sort of solution and think they can get it to solve all of their problems.

Hadoop is a good example, lots of poor assumptions are made around Hadoop. It’s designed to do one thing really well, and it does that fairly well. But when you think you can adapt it into a more general purpose storage system it starts falling apart. Which is completely understandable, it wasn’t designed for that purpose. Many people don’t understand that simple concept though.

Another poor use of Hadoop is trying to shoehorn a real time application on top of it, it just doesn’t work. Yet there are people out there (I’ve talked to some of them in person) who have devoted significant developer resources to try to attack that angle. Spend thirty minutes of time researching the topic and you can realize pretty quickly that it is a wasted effort. Google couldn’t even do it!

Speaking of Hadoop, and Oracle for that matter it seems Oracle announced a Hadoop-style system yesterday at Open World, only Oracle’s version seems to be orders of magnitutde faster (and more orders of magnitude expensive given the amount of flash it is using).

Using the skinnier and faster SAS disks, Oracle says that the Exadata X2-8 appliance can deliver up to 25GB/sec of raw disk bandwidth on uncompressed data and 50GB/sec across the flash drives. The disks deliver 50,000 I/O operations per second (IOPs), while the flash delivers 1 million IOPs. The machine has 100TB of raw disk capacity per rack and up to 28TB of uncompressed user data. The rack can load data at a rate of 5TB per hour. Using the fatter disks, the aggregate disk bandwidth drops to 14GB/sec, but the capacity goes up to 336TB and the user data space grows to 100TB.

The system is backed by an Infiniband-based network, I didn’t notice specifics but assume 40Gbps per system.

Quite impressive indeed. Like Hadoop, this Exadata system is optimized for throughput, it can do IOPS pretty well too but it’s clear that throughput is the goal. By contrast a more traditional SAN gets single digit gigabytes per second even on the ultra high end for data transfers at least on the industry standard SPC-2 benchmark.

  • IBM DS8700 rated at around 7.2 Gigabytes/second with 256 drives and 256GB cache costing a cool $2 million
  • Hitachi USP-V rated at around 8.7 Gigabytes/second with 265 drives and 128GB cache costing a cool $1.6 million

Now it’s not really apples to apples comparison of course, but it can give some frame of reference.

It seems to scale really well according to Oracle –

Ellison is taking heart from the Exadata V2 data warehousing and online transaction processing appliance, which he said now has a $1.5bn pipeline for fiscal 2011. He also bragged that at Softbank, Teradata’s largest customer in Japan, Oracle won a deal to replace 60 racks of Teradata gear with three racks of Exadata gear, which he said provided better performance and which had revenues that were split half-and-half on the hardware/software divide.

From 60 to 3? Hard to ignore those sorts of numbers!

Oh and speaking of Facebook, and Hadoop, and Oracle, as part of my research into the topic of Hadoop I came across this, I don’t know how up to date it is but thought it was neat. Oracle DB is one product I do miss using, the company is filled with scumbags to be sure, I had to educate their own sales people on their licensing the last time I dealt with them. But it is a nice product, works really well, and IMO at least it’s pretty easy to use especially with enterprise manager (cursed by DBAs from coast to coast I know!). Of course makes MySQL look like it’s a text file based key-value pair database by comparison.

Anyways onto the picture!

Oh my god! Facebook is not only using Hadoop, but they are using MySQL, normal NAS storage, and even Oracle RAC! Who’da thunk it?

Find a tool or a solution that does everything well? The more generic the approach, the more difficult it is to pull it off, which is why so many solutions like that typically cost a significant amount of money, because there is significant value in what the product provides. If perhaps the largest open source platform in the world (Linux) has not been able to do it (how many big time open source advocates do you see running OS X and how many run OS X on their servers), who can?

That’s what I thought.

(posted from my Debian Lenny workstation with updates from my Ubuntu Lucid Lynx laptop)

September 7, 2010

We need a new theme

Filed under: General,Random Thought — Tags: — Nate @ 11:42 pm

Do you know WordPress? Good because I sure as hell don’t.

We need a new theme, can you give us some suggestions? My main complaint about the theme we have now is that it doesn’t make effective use of screen real estate for larger resolutions. I mean I feel like I’m stuck in the 90s when viewing our page at 1080p resolutions. Though with a firefox zoom plugin it’s more usable, I have to zoom it in 170%., even then there’s quite a bit of dead space.

Beyond that just something that is pretty simple I guess? I don’t know, none of us are web developers I don’t think so we aren’t able to customize it or whatever.

August 15, 2010

Lowest power dual socket server ever

Filed under: Datacenter,General — Tags: , , — Nate @ 12:20 pm

This was posted a couple of weeks ago but I was on vacation at the time and didn’t notice it until a few days ago.

It talks about the latest 4000-series low power chips from AMD running in a dual socket system from ZT Systems.

The numbers are pretty startling. At peak load they measure the power draw at only 126 watts for the system as a whole:

  • Dual processor 6-core Opteron 4164 EE (1.8Ghz per core)
  • 16GB memory (4x4GB DDR3-1333)
  • 128GB SSD

From the blog:

[..] There are four major enhancements to the AMD Opteronâ„¢ 4000 Series platform which significantly lower server power consumption:

  1. The AMD Opteron™ 4100 EE Series of processors are the lowest power AMD Opteron processors ever. These processors are rated at 32W ACP, which is 20% lower than AMD’s previous generation 2400 EE Series processors.
  2. AMD Opteronâ„¢ 4100 Series processors support 1.35V DDR3 memory, enabling lower server power consumption at load.
  3. The AMD Opteronâ„¢ 4000 Series platform uses low-power chipsets. The SR5650 has a maximum TDP of only 13 watts.
  4. AMD Opteron™ 4100 Series processors include new AMD-P power management features, including C1E. C1E is a feature that helps reduce the power consumption of the AMD Opteron™ 4100 Series processor’s integrated memory controller and HyperTransport™ technology links.

[..]
The two lowest power Intel Xeon processor-based servers consume 28% more and 34% more power than the ZT Systems 1253Ra Datacenter Server[..]

Pretty amazing that you can get a dual processor, 12 core(total) system running at less power than some CPUs out there consume by themselves.

I’m sure it will run even at even lower power with rack level DC power and cooling.

August 7, 2010

Good Gog

Filed under: General — Nate @ 11:26 am

Almost done with my vacation, I have been reading this book recommended by a friend that has a lot of different stories in it from different time periods, and during reading it I thought of a couple games I used to play a long time ago – Castles, and Castles II, with the first one being almost 20 years old at this point. I wanted to play it again, so I searched around and pretty quickly came across a pretty cool site – Gog. Very professionally done, they seem to package everything you need for some of these really old games at a low price. Including integrated DOSBox, installation, even the manuals are included.

I don’t know if they are legit or not, whether or not the original publishers have given their blessing to re-distribute these games(I would expect not), but there is likely very few legit ways to acquire these games at this point anyway. But for $6 to get Castles and Castles II in a single package was more than worth it. I spent nearly two full days playing these games, had a lot of fun.

If  you know me, you know I don’t play a lot of games these days, so I was happy to come across this on vacation so I could devote some real time to them, and I beat them both pretty quickly so I won’t have it in the back of my mind that I need to play them more in the coming weeks. Though I did come across several other games that look neat too. I also bought Descent & Descent II from Gog, I had good memories of Descent II, I was really good at that game back in the day. I haven’t downloaded it yet though.

I’m not going to vouch for Gog, though so far the experience has been good and the result very high quality. There are a lot of key games that are missing from Gog still, so I’m not sure if they just don’t have all the materials yet, or if they are legit perhaps the publishers have denied them rights or something. I’m thinking of the Wing Commander series and the X-wing series. I did go through and re-buy most of the Wing Commander games last year, it really feels strange to me as someone who likes modern technology to find myself really enjoying playing some games that I used to play on a 286 15+ years ago(in the case of Castles, and Wing Commander I any ways).

While I’ve been away from home I did notice that the FTC settled with Intel (in an update to an earlier posting), time will tell if it does anything, I am in agreement with this statement from The Register reporter:

To be even more honest, you could argue that the damage to AMD has already been done and that a resurgent Intel got away with all kinds of anticompetitive practices. It is debatable as to whether AMD, VIA, and Nvidia (which may now enter the x86 chip market) can beat Intel even on a more-level playing field, now that Intel has its engineering act together.

So on that note..

Happy Saturday from Cannon Beach Oregon.

July 16, 2010

Moving on up, to the west side

Filed under: General — Nate @ 7:40 am

So I made the decision, I gave my notice to quit my current gig a couple of days ago. Out of respect for most of my fellow co-workers, I offered to stay the standard two weeks to transition some stuff to other people. I could leave tomorrow and be fine (or yesterday for that matter), doesn’t really matter to me.

I just wanted to thank all of my friends, co-workers, acquaintances, recruiters and those I have interviewed with over the recent weeks for the positive support you have given me. It really helped make the decision much easier. I’ve worked hard, harder than most people I know work in their careers to get where I am today. I’ve sacrificed a lot from a personal level, because I didn’t want to get caught off guard like I did during the last recession. And when I saw signs the economy and society in general was peaking, it was more motivating to get to the top of my game and stay there. And I believe I’m there now(have been for a while). Really takes the stress of a down economy away.

I went through four different job offers before settling on a “Web 3.0” startup in Seattle. They seem to be growing fast and in great need for someone like me. This was the best and most active job search I’ve had in my career, I’ve turned down companies before but I’ve never been so pursed by them both before and after I turned them down. It feels so encouraging that there is such a high demand for folks such as myself, that made the decision even easier — if I don’t have to put up with the current situation that has been festering for years (before I started even!) with no end in sight then I don’t have to. It’s a very fortunate position to be in I have no problem admitting that, I know people who are in, how should I say it, less than ideal career situations but aren’t as fortunate to be able to find a more healthy work atmosphere at another organization.

My work is my life( I’m working on fixing that still), so having a positive professional experience on the job is very important to me. Becuase my work is my life, myself I know of nobody more motivated, more dedicated to their work at least in my field. Which makes it that much harder when an employment situation doesn’t work out.

Anyways, on to the category of jobs, my new gig is hiring many more people, including what appears to be a mid level linux system administrator, if you are looking to make a move, drop me a line with your resume and I’ll pass it along. I would expect the position to have career growth opportunities given the size of the company and where they are at (they are a pretty young company but not your 10-20 person startup).

Other positions include a lead PHP developer, and a Java developer, I can send you more complete job descriptions if you are interested. And of course willing to split the referral bonus 🙂

One thing I have learned though, is think twice(or three times) when the position your interviewing for was for someone else who left suddenly. There may be very good reasons why they left, or maybe not. I first noticed signs of problems more than a year and a half ago, but decided to try to work through them and see if things could work out, no such luck! My current position is my 3rd position in which I replaced someone else. The first was my first job, the previous guy was a flake(showed up to work maybe 25% of the time), the second I forgot why the previous person left but the company was halfway decent and I got a lot of stuff accomplished until the economy ate them. And my current is of course the same situation, and the previous two people left for much of the same reason I am, in fact I’m told that I have held my position longer than anyone else had held it at the company. What does that tell you?

Oh and in case it wasn’t obvious, my current job is probably open, I can’t reccomend anybody senior going there, for reasons I won’t get into here. But if your a junior or mid level admin without a lot of career aspirations then it could be a good fit.

June 18, 2010

Cross Pollination

Filed under: General — Nate @ 8:44 am

I don’t know about you but to me it’s kind of odd. It’s not something I would expect to be such a regular occurrence. Maybe it’s just the region, or maybe it’s broader.

I’m talking about vendor sales reps and engineers jumping from one company directly to the competition. I mean just from my own experience, I know people who have

  • Moved from Extreme networks to Juniper
  • Moved from Foundry networks to Extreme
  • Moved from HP networking to Extreme
  • Moved from Foundry networks to A10 networks
  • Moved from F5 networks to A10 networks
  • Moved from EMC to 3PAR
  • Moved from NetApp to 3PAR
  • Moved from Equallogic to 3PAR
  • And recently saw someone I hadn’t  seen in more than a year and a half moved from Hitachi to Xiotech

Maybe it’s just the line of work, but to me at least it seems like an amazing amount of cross pollination to the point where it’s hard to tell on occasion what the person really believes, I mean one minute they are pitching product X from you and bashing product Y then the next their doing the opposite.

Then there are other less direct migrations of course going from a manufacturer to a VAR or a distributor, but I’ve been more fascinated by those making the leap from one manufacturer to another.

I suppose it’s just a job at the end of the day.

May 29, 2010

Tapped out

Filed under: General,News — Nate @ 12:20 pm

An interesting article on slashdot posted in reference to water shortages and companies like Intel and Coke consuming large amounts of water that might otherwise be used for farming.

About 2.4 billion people live in “water-stressed” countries such as China, according to a 2009 report by the Pacific Institute, an Oakland, California-based nonprofit scientific research group

[..]

China’s 1.33 billion people each have 2,117 cubic meters of water available per year, compared with 1,614 cubic meters in India and as much as 9,943 cubic meters in the U.S., according to the Food and Agriculture Organization of the United Nations.

Nothing new really if you have been paying attention for the past few years. I really try very hard to keep this blog as technical as possible no matter how strong my emotions are to rant against the government and society in general, in this case I’ll venture a bit outside of the technical realm thanks to the above article mentioning Intel’s water intensive business.

Another water intensive business is data centers, perhaps one of the more extreme examples is the SuperNap outside Las Vegas, where one person is quoted as saying it will require millions of gallons of water per day:

“They’re in the middle of the desert and will need almost 3 million gallons of water per day for blowdown and evaporation for their 30,000 ton evaporative cooling plant.”

While I can’t vouch for the sources, just take a look at where some people think we are headed as far as global population growth is concerned, and notice similar trend lines from those that are in the global warming camp, and even more similar trend lines from those reporting on U.S. debt.

I’ll end the tangent here, but you can probably get an idea of where our civilization is headed.

May 28, 2010

Google out of gas?

Filed under: General — Nate @ 9:26 pm

I don’t know if it’s true or not, I certainly hope so. Indications point to them being out of gas as far as their big growth earlier this decade. I watched a report on CNBC about a week or two ago where a couple of analysts agreed that Google is going nowhere fast. I don’t pay that close attetion to them with whatever products they launch etc. But do feel a bit more at ease if Google has infact peaked. I know a lot of people believe that they pulled out of China to save face because they had trouble competing, which certainly sounds like a more plausable explanation than their official excuse.

It’s been a while since I blogged on anything and had this in my head since I saw it, so wanted to mention the report.

April 30, 2010

Violate Electrical specs for more servers?

Filed under: Datacenter,General,Random Thought — Tags: — Nate @ 8:46 pm

As usual on big blog posts I often literally go back and re-read the post about 60 times and think about what I wrote.

Well I was reading my most recent post about Datacenter Dynamics specifically the stranded power section and the datacenter operators of hyperscale facilities wanting to draw every watt they can off the circuits to maximize efficiency and I got to thinking..

Would they go so far as to violate electrical specs by drawing more than 80% of the power for a particular circuit? I mean in theory at least if they construct the components properly they can probably do it fairly safely. I learned a few years ago from someone, that the spec in question is NEC Section 384-16(c). Which I think in part reads:

The NEC requires the branch circuit computed load for conductor sizing to be sized at 125% of the continuous load, plus the noncontinuous load (100%).

Which equates to 80% utilization. If you know your power usage levels that well, and your loads etc, do you think such hyperscale facilities would run at 85%? 90%? 95% of circuit load? Really with all of the other extreme measures being taken to maximize efficiency I wouldn’t put it past them. They’re going so far as to design special motherboards and have specific components down to the VRMs to lower power usage. I can see them investing in higher grade electrical gear allowing them to safely operate at higher circuit draws, especially when you take into account power capping as well. Afterall, if your spending the effort to shave literally single digit watt usage off your systems that extra 20% capacity on the circuit has to be very tempting to use.

I remember a few years ago doing a load test on one of the aforementioned lower quality power strips(they weren’t cheap, but the company’s QA wasn’t up to par), it was a 30A PDU. And I loaded it up with a bunch of systems, and walked away for a couple minutes and came back and was shocked to see the meter reporting 32A was being drawn. I immediately yanked some of the power cords out to get it back under 30A. After talking with the manufacturer (or maybe it was another manufacturer I don’t recall), they said that was not unexpected, the breaker has some sort of internal timer that will trip based on the amount of excess load on the circuit, so if your drawing 30A it probably won’t trip for a while, if your drawing 32A then it may trip after a few minutes, if you try to draw 40A it will likely trip immediately(I’m guessing here).

March 17, 2010

Frightened

Filed under: General,Networking — Tags: — Nate @ 8:15 pm

Frightened. That was the word that first came to my mind when I read this article from our friends at The Register.

The report also says that 60 per cent of Google’s traffic is now delivered directly to consumer networks. In addition to building out a network of roughly 36 data centers and co-locating in more than 60 public exchanges, the company has spent the past year deploying its Google Global Cache (GGC) servers inside consumer networks across the globe. Labovitz says that according to Arbor’s anecdotal conversations, more than half of all consumer providers in North American and Europe now have at least one rack of Google’s cache servers.

Honestly, I am speechless beyond the word frightened, you may want to refer to an earlier blog post “Lesser of two Evils” for more details.

« Newer PostsOlder Posts »

Powered by WordPress