TechOpsGuys.com Diggin' technology every day

August 9, 2011

How hard is it to run a mail server

Filed under: Random Thought — Tags: — Nate @ 11:23 pm

I read slashdot quite often, mostly for the comments, I post (as AC) maybe once every 3 years, but find the discussions interesting on occasion.

One such discussion was here, where someone was asking for advice as to how best to migrate off of gmail onto their own hosted platform. To me it seemed simple enough, but honestly I could not believe the negative response towards running your own mail server.

First off I’ll say I haven’t run a “corporate” mail server for almost a decade now, I have run several mail relays for companies for applications and stuff. I have been running my own mail server for my own personal (and some family) use for more than a decade, and I run another mail server that has maybe a dozen people on it, left overs from when I ran a small ISP in Washington.

So nothing major. I didn’t get the impression that the poster on slashdot was asking for anything major. But I was seeing people talk about massive headaches with blacklisting, anti spam, having to worry about disaster recovery, data replication, and the constant hand holding and patching of the system to keep it running.

I just didn’t get it. I mean sure it took some effort to set up the system I have which is pretty basic, it really requires minimal maintenance, I have never been blacklisted, really have minimal spam problems (very manageable anyways for me).

My setup is basic as I mentioned:

  • Postfix for SMTP – I setup quite a bit of anti spam stuff many, many years ago but really haven’t touched it much at all since.
  • SpamAssassin – for – duh – spam. I took some time to integrate this into incoming postfix email and it flags messages as **** SPAM ****  in the subject when something hits the spam filter, I have server side mail filters that move that to a dedicated folder. In all my years I have never noticed a false positive and have never had anyone complain that they can’t email me for a reason related to Spam Assassin flagging their email as Spam. My biggest potential issue with Spam Assasin is I probably get 150 spam (that get past the filters) for every real email I get (I don’t get a lot of email at home excluding mailing lists that I occasionally participate in). So I don’t have much “ham” to train SA with. I haven’t recently tried to determine how much spam is blocked at the various levels but last time I did (many years ago) it was quite a bit.
  • Anomy Sanitizer – this does quite a few things such as stripping HTML email, stripping bad attachments etc. I’m sure it goes overboard in a lot of cases, and most users probably wouldn’t like it, stripping HTML email probably causes the most usability issues for me as some emails don’t come in with plain text as well as HTML, so some times I  get email that says “Hey click on this link to unsubscribe (or do some other action – e.g. rate Netflix quality back when I used their stuff)” only to find Sanitizer stripped the html so there is no link to click on, and no url I can copy/paste to the browser. But IMO at least it’s a small price to pay
  • Cyrus IMAP 2.1 for IMAP – I started using Cyrus back in 2000 when I migrated a company off of UW IMAP onto Cyrus because it had some more advanced functionality vs Courier at the time (don’t remember what). I’ve stuck to it because it seems to work for me. I create different email addresses for pretty much every organization I deal with and have those go to dedicated IMAP folders (server side – not using filtering, postfix delivers directly to the mailbox), so most of the time I am unsubscribed to 85% of the IMAP “user accounts”, and only subscribe when I need to, email collects silently in the background in the meantime. User accounts is in quotes because I use a single account to access all of the other IMAP accounts (which can be problematic for some email clients because they make certain assumptions that don’t apply to me)
  • Squirrelmail for webmail – The UI is basic, hasn’t really changed much since – you guessed it – about 2000 when I first started using it, I have, on occasion looked for alternatives but have not found one (until this slashdot discussion that is) that looked interesting enough to try. One big feature I like about squirrelmail is the ability to have many, many “From” email addresses, and select them from a drop down box. I have upwards of 100 different email accounts(view from the outside world – from my view it’s a single account), I maybe need to send mail “From” from maybe 10-20. So when I compose an email I select which email address to send “From”. Squirrelmail is smart enough if I hit reply on an email someone sent, say to my linkedin@ address, the mail client will automatically select the right “From” email address to use without me having to think about it. Since Sanitizer strips out html from emails I don’t believe I have to worry about XSS bugs in Squirrelmail because it’s all stripped out (but I could be wrong I’m not sure). As a result I haven’t upgraded in eons..

The last time I did major changes to my system was probably 5-6 years ago, those were introducing Spam Assassin to the system, and the more painful process of upgrading from Cyrus 1.x to 2.x (it was an ugly upgrade process).

I don’t use anti virus, never needed it(I integrated anti virus with Sendmail at the one company that I did manage the corporate email servers for back in 2000-2002). A lot of my habits and practices were set up a long time ago and there may very well be better ways to go about things these days(one thing would be to investigate using spamd for anti spam instead of spawning a separate SA process for each message), but what I have works, it doesn’t cause issues, I just don’t understand what some of those people were complaining about when running their own personal mail server.

I also don’t do any sort of calendaring system – never really needed it for personal use.

Sure it requires some setup, and you need to be smart (forward & reverse DNS, you want clean IPs that aren’t blacklisted (easy ways to check that are out there). I saw one guy say the person should get a block of 30-40 IPs and put the mail server in the middle of the block! I mean are you kidding me?

Even back when I ran an ISP with maybe 50-60 users (yes it was a small ISP – back in late 90s mostly), we never had blacklisting or spam problems. Maybe we were lucky I don’t know.

I just couldn’t believe the experiences some of these people were posting. Sure I can understand having those kinds of issues if your running a big mail system for a lot of users, but the impression I got was the original poster was looking to run a setup sort of like mine – a small number of users – or maybe just themselves.

I’ve even gone through the process of migrating my mail system from:

  • Office with dedicated T1 lines to..
  • Home with DSL connection to ..
  • Temporary office with T1 lines while my DSL is moved to ..
  • Home with DSL connection to ..
  • Co-location to..
  • Terremark Cloud (where it runs now) and this weekend to..
  • Co-location

And all of the DNS changes and stuff associated with it, for the most part the migrations are pretty painless.

A couple of my users were wondering how I went about moving all of their data and stuff from Washington to Miami in a matter of minutes without them noticing anything, it takes some moderate planning but in the grand scheme of things it’s not that difficult from a technical perspective. Next weekend I’ll be migrating them again from Miami to California – their data resides on a different system from mine.

I just wanted to write about my experience – sure it does take some work – depending on how much functionality  you want  – to initially set  up the system. But in my experience once it is setup, the amount of work to maintain it is minimal.

I like the privacy and control I get with running my own stuff. I sure as hell don’t trust google with my data, they could pay me $100/mo for hosting my email with them and I wouldn’t do it. But for others it may be a good option.

I did use Gmail at the last company I worked at, their corp email was Gmail. I really didn’t like it – but what surprised me the most was how slow the search was. I was expecting anything I searched for would be returned in a fraction of a second but it took much longer (not forever but 10-20x longer than I was originally expecting). I never got used to how they organize their mail, with the tags and stuff. Even after using it for ~8 months I never warmed up to it, probably because there was mini demons in the back of my head screaming at me not to like it because I don’t trust Google, I’m sure that had something to do with it.

But for others maybe it’s the best way, or hotmail, or yahoo, or whatever.. I’ve been hosting my own email for so long I never really used anything else.

The most annoying problem I think I ever came across running my own mail system was not granting Spam Assassin enough memory – sometimes it would puke causing the email to get garbled – it would happen maybe once every 150,000 emails or something (which means maybe 2-3 times per year), I ignored it for a few years finally decided to look into it and found that SA was running out of memory so I gave it more(I think I went from something like granting it 32MB to 128MB), hasn’t happened since.

Do I have disaster recovery? No – don’t need it. If my server goes down for a few hours or a day or whatever, mail is likely queued at the other end (mail is often queued for up to what – 5 days before being dropped from the queue?). What happens if I lose an email? Really not the end of the world. I have had a few times when people say my system rejected their message – and sometimes it does, I wrote (again a long time ago) a lot of regular expression checks to try to detect spam, and sometimes it gets a false positive, so I fix it and move on, it’s rare though(again a few times a year at most). If the email is THAT CRITICAL then if they really can’t get through to me they’ll call. And if I don’t answer(or don’t return the call) and it’s THAT CRITICAL – they’ll call again later.

Of course my co-location/cloud stuff doesn’t run just email – it runs this blog, my basic web sites, DNS, and my new co-location serves as my off site backup with ~3.5TB of usable disk space on the system, I brought the system home last weekend and sync’d up 1.7TB of data to it.

Moral of the story is – if you really want to run your own mail system, don’t be afraid – it’s not THAT hard.

August 3, 2011

VMware revamps vSphere 5 licensing again

Filed under: Virtualization — Tags: , — Nate @ 5:40 pm

I guess someone over there high up was listening, nice to see the community had some kind of impact, VMware has adjusted their policies to some degree, far from perfect, but more bearable than the original plan.

The conspiracy theorist makes me think VMware put bogus numbers out there to begin with, never having any intension of following through with them to gauge the reaction, and then adjusted them to what they probably originally would of offered and try to make people think like they “won” by getting VMware to reduce the impact to some degree.

vSphere Enterprise List Pricing comparison (w/o support)

# of SocketsRAMvSphere 4 EnterprisevSphere 5
Enterprise
(old)		vSphere 5
Enterprise
(new)		Cost increase over vSphere 4
2256GB2 Licenses - $5,7508 Licenses - $23,0004 Licenses - $11,500100%
4512GBN/A16 Licenses - $46,0008 Licenses - $23,000N/A
81024GBN/A32 Licenses - $92,00016 Licenses - $46,000N/A

vSphere Enterprise+ List Pricing comparison (w/o support)

# of SocketsRAMvSphere 4 Enterprise+vSphere 5 Enterprise+
(old)		vSphere 5 Enterprise+
(new)		Cost increase over vSphere 4
2256GB2 Licenses - $6,9905 Licenses (240GB) - $17,4753 Licenses (288GB) - $10,48550% higher
4512GB4 Licenses - $13,98011 Licenses (528GB) - $38,4455 Licenses (480GB) - $17,47525% higher
81024GB8 Licenses - $27,96021 Licenses(1008GB) - $73,995
11 Licenses (1056GB) - $38,44537% higher

There were other changes too, see the official VMware blog post above for the details. They quadrupled the amount of vRAM available for the free ESXi to 32GB which I still think is not enough, should be, say at least 128GB.

Also of course they are pooling their licenses so the numbers fudge out a bit more depending on the # of hosts and stuff.

One of the bigger changes is VMs larger than 96GB will not need more than 1 license. Though I can’t imagine there are many 96GB VMs out there… even with 1 license if I wanted several hundred gigs of ram for a system I would put in on real hardware, get more cpu cores to boot (not unlikely you have 48-64+ cores of cpu for such a system, which is far beyond where vSphere 5 can scale to for a single VM).

I did some rounding in the price estimates, because the numbers are not divisible cleanly by the amount of ram specified.

It seems VMware has effectively priced their “Enterprise” product out of the market if you have any more than a trivial amount of memory. vSphere 4 Enterprise was, of course limited to 256GB of ram, but look at the cost of that compared to the new stuff, pretty staggering.

Quad socket 512GB looks like the best bet on these configurations anyways.

I still would like to see pricing based more on features than on hardware.  E.g. give me vSphere standard edition with 96GB per CPU of vRAM licensing, because a lot of those things in Enteprise+ I don’t need (some are nice to have but very few are critical for most people I believe). As-is users are forced into the higher tiers due to the arbitrary limits set on the licensing, not as bad as the original vSphere 5 pricing but still pretty bad for some users when compared to vSphere 4.

Or give me free ESXi with the ability to individually license software features such as vMotion etc on top of it on a per-socket basis or something.

I think the licensing scheme needs more work. VMware could also do their customers a favor by communicating how this will change in the future, as bigger and bigger machines come out it’s logical to think the memory limits would be increased over time.

The biggest flaw in the licensing scheme remains it measures based on what is provisioned, rather than what is used. There is no excuse for this from VMware since they own the hypervisor and have all the data.

Billing based on provision vs usage is the biggest scam in this whole cloud era.

July 25, 2011

Netflix acknowledges significant customer backlash

Filed under: General — Tags: , — Nate @ 5:16 pm

Was watching some CNBC recently and they were talking about the upcoming Netflix results and how much of an impact their recent price hikes may cause.

I wondered over to Yahoo! and came across this:

SAN FRANCISCO (AP) — Netflix Inc. is bracing for customer backlash that could result in its slowest subscriber growth in more than three years amid changes to its online video and DVD rental service that will raise prices by as much as 60 percent.

[..]

The shortfall stems from an anticipated slowdown in Netflix’s subscriber growth amid the most radical change in the company’s pricing since it began renting DVDs through the mail 12 years ago.

Nice to see. I don’t blame Netflix for the price hikes, I didn’t like them so I quit the service, but it seems clear they are losing money pretty badly (apparently they’ve been using fancy accounting things to try to cover this up), and their licensing costs are about to skyrocket.

Netflix spent nearly $613 million on streaming rights in the second quarter, a more than nine-fold increase from the same time last year. The company so far has signed long-term contracts committing it to pay $2.44 billion for streaming rights.

So they’re doing what they have to do. Though I’m sure most everyone agrees they could of handled the situation far better than they did. They also apparently face some stiff competition in the latin america markets where they are expanding to, places where bandwidth pipes are smaller(making streaming less feasible), and cable bills are much cheaper than they can be here in the states.

While Netflix’s price hikes have gotten quite a bit of press at least in the business news recently I am kind of surprised that the same hasn’t seemed to be true of the VMware price hikes (outside of the tech community at least). The outrage continues to build..

For me it all comes down to selection – increase the streaming catalog to at least match whatever they have on DVD now and I would probably jump back on board.. in the mean time I’ll stick to cable(+Tivo), I’ll pay more but I get a lot more value out of it.

July 20, 2011

I called it! – Force10 bought by Dell

Filed under: Networking — Tags: , — Nate @ 11:35 am

Not that it matters to me too much either way but Dell just bought Force10. I called it! Well it matters to me in that I didn’t want Dell near my Extreme Networks 🙂

It is kind of sad that Force10 was never able to pull off their IPO. I have heard that they have been losing quite a bit of talent recently, but don’t know to what degree. It’s also unfortunate they weren’t able to fully capitalize on their early leadership in the 10 gigabit arena, Arista seems to be the new Force10 in some respects, though it wouldn’t surprise me if they have a hard time growing too barring some next gen revolutionary product.

I wonder if anyone will scoop up BlueArc, they have been trying to IPO as well for a couple of years now, I’d be surprised if they can pull it off in this market.  They have good technology just a whole lot of debt. Though recently I read they started turning a profit..

 

VMware Licensing models

Filed under: Virtualization — Tags: , — Nate @ 5:38 am

[ was originally combined with another post but I decided to split out ]

VMware has provided it’s own analysis of their customers hardware deployments and telling folks that ~95% of their customers won’t be impacted by the licensing changes. I feel pretty confident that most of those customers are likely massively under utilizing their hardware. I feel confident because I went through that phase as well. Very, very few workloads are truly cpu bound especially with 8-16+ cores per socket.

It wouldn’t surprise me at all that many of those customers when they go to refresh their hardware change their strategy pretty dramatically – provided the licensing permits it. The new licensing makes me think we should bring back 4GB memory sticks and 1 GbE. It is very wasteful to assign 11 CPU licenses to a quad socket system with 512GB of memory, memory only licenses should be available at a significant discount over CPU+memory licenses at the absolute minimum. Not only that but large amounts of memory are actually affordable now. It’s hard for me to imagine at least having a machine with a TB of memory in it for around $100k, it wasn’t TOO long ago that it would of run you 10 times that.

And as to VMware’s own claims that this new scheme will help align ANYTHING better, by using memory pools across the cluster – just keep this in mind. Before this change we didn’t have to care about memory at all, whether we used 1% or 95%, whether some hosts used all of their ram and others used hardly any. It didn’t matter. VMware is not making anything simpler. I read somewhere about them saying some crap about aligning more with IT as a service. Are you kidding me? How may buzz words do we need here?

The least VMware can do is license based on usage. Remember pay for what you use, not what you provision. When I say usage I mean actual usage. Not charging me for the memory my Linux systems are allocating towards (frequently) empty disk buffers (goes to the memory balloon argument). If I allocate 32GB of ram to a VM that is only using 1GB of memory I should be charged for 1GB, not 32GB. Using vSphere’s own active memory monitor would be an OK start.

Want to align better and be more dynamic? align based on memory usage and CPU usage, let me run unlimited cores on the cluster and you can monitor actual usage on a per-socket basis, so if on average (say you can bill based on 95% similar to bandwidth) your using 40% of your CPU then you only need 40% licensing. I still much prefer the flat licensing model in almost any arrangement rather than usage based but if your going to make it usage based, really make it usage based.

Oh yeah – and forget about anything that charges you per VM too (hello SRM). That’s another bogus licensing scheme. It goes completely against the trend of splitting workloads up into more isolated VMs and instead favors fewer much larger VMs that are doing a lot of things at the same time. Even on my own personal co-located ESXi server, I have 5 VMs on it, I could consolidate it to two and provide the similar end user services, but it’s much cleaner to do it in 5 for my own sanity.

All of this new licensing stuff also makes me think back to a project I was working on about a year ago, trying to find some way of doing DR in the cloud, the ROI for doing it in house vs. any cloud on the market(looked at about 5 different ones at the time) was never more than 3 months. In one case the up front costs for the cloud was 4 times the cost for doing it internally. The hardware needs were modest in my opinion, with the physical hardware not even requiring two full racks of equipment. The #1 cost driver was memory, #2 was CPU, storage was a distant third assuming the storage that the providers spec’d could meet the IOPS and throughput requirements, storage came in at about 10-15% of the total cost of the cloud solution.

Since most of my VMware deployments have been in performance sensitive situations (lots of Java) I run the systems with zero swapping, everything in memory has to stay in physical ram.

Cluster DRS

Filed under: Virtualization — Tags: , — Nate @ 12:05 am

Given the recent price hikes that VMware is imposing on it’s customers(because they aren’t making enough money obviously) , and looking at the list of new things in vSphere 5 and being, well underwhelmed (compared to vSphere 4), I brain stormed a bit and thought about what kind of things I’d like to see VMware add.

VMware seems to be getting more aggressive in going after service providers (their early attempts haven’t been successful, it seems they have less partners now than a year ago – btw I am a vCloud express end-user at the moment). An area that VMware has always struggled in is scalability in their clusters (granted such figures have not been released for vSphere 5 but I am not holding my breath for a 10-100x+ increase in scale)

Whether it’s the number of virtual machines in a cluster, the number of nodes, the scalability of the VMFS file system itself (assuming that’s what your using) etc.

For the most part of course, a cluster is like a management domain, which means it is, in a way a single point of failure. So it’s pretty common for people to build multiple clusters when they have a decent number of systems, if someone has 32 servers, it is unlikely they are going to build a single 32-node cluster.

A feature I would like to see is Cluster DRS, and Cluster HA. Say for example you have several clusters, some clusters are very memory heavy for loading a couple hundred VMs/host(typically 4-8 socket with several hundred gigs of ram), others are compute heavy with very low cpu consolidation ratios (probably dual socket with 128GB or less of memory). Each cluster by itself is a stand alone cluster, but there is loose logic that binds them together to allow the seamless transport of VMs between clusters either for either load balancing or fault tolerance. Combine and extend regular DRS to span clusters, on top of that you may need to do transparent storage vMotion (if required) as well along with the possibility of mapping storage on the target host (on the fly) in order to move the VM over (the forthcoming storage federation technologies could really help make hypervisor life simpler here I think).

Maybe a lot of this could be done using yet another management cluster of some kind, a sort of independent proxy of things (running on independent hardware and perhaps even dedicated storage). In the unlikely event of a catastrophic cluster failure, the management cluster would pick up on this and move the VMs to other clusters and re start them (provided there is sufficient resources of course!). In very large environments it is not be possible to map everything to everywhere, which would require multiple storage vMotions in order to get the VM from the source to a destination that the target host can access – if this can be done at the storage layer via the block level replication stuff first introduced in VAAI that could of course greatly speed up what otherwise might be a lengthy process.

Since it is unlikely anyone is going to be able to build a single cluster with shared storage that spans a great many systems(100s+) and have it be bulletproof enough to provide 99.999% uptime, this kind of capability would be a stop gap, providing the flexibility and availability of a single massive cluster, while at the same time reducing the complexity in having to try to build software that can actually pull the impossible (or what seems impossible today) off.

On the topic of automated cross cluster migrations, having global spare hardware would be nice too, much like most storage arrays have global hot spares, which can be assigned to any degraded RAID group on the system regardless of what shelf it may reside on. Global spare servers would be shared across clusters, and assigned on demand. A high end VM host is likely to cost upwards of $50,000+ in hardware these days, multiply by X number of clusters and well.. you get the idea.

While I’m here, I might as well say I’d like the ability to hot remove memory, Hyper-V has dynamic memory which seems to provide this functionality. I’m sure the guest OSs would need to be re-worked a bit too in order to support this, since in the physical world it’s not too common to need to yank live memory from a system. In the virtual world it can be very handy.

Oh and I won’t forget – give us an ability to manually control the memory balloon.

Another area that could use some improvement is the vMotion compatibility, there is EVC, but last I read you still couldn’t cross processor manufacturers when doing vMotion with EVC. KVM can apparently do it today.

July 19, 2011

Something I hope I don’t miss – Seattle Weather

Filed under: General — Tags: — Nate @ 11:09 pm

I’ve been seeing/hearing increasing numbers of folks in the area complaining about an apparent lack of summer round these parts. Myself I welcome the cooler weather, in fact despite the cooler weather I have had my ACs running almost non stop for at least a couple months now (compressors weren’t always running) with a target temp of 68 degrees.

Actually shortly after I moved here, I was watching a local newscast on one of the channels, and the weather guy was complaining about the weather every chance he got, apparently he hated being here, after a couple months I stopped watching them.

I never minded the rain here, much of my life I have spent time where there was roughly equal if not more annual rainfall than is here. I don’t mind gray skies, doesn’t really matter to me.

People tell me it’s less about the total amount of rain and more about how spread out throughout the year it is. I guess I don’t get out enough to notice.

I have been casually keeping track of the weather of where I am moving to, to see how much of an adjustment I may have ahead of me as a result of the move, after all I am moving roughly 850 miles to the south.

For the most part the temperature seems to be about the same, at least since I started looking in June. My new place is fairly close to both the Pacific Ocean and the San Fransisco bay which helps keep it cooler than if I was more inland.

I just got done watching yet another news report on people complaining about how cold it is here (61 degrees at the moment at 11PM). My own ideal temperature is highs in the mid to upper 50s, lows in the mid 40s (mainly because my apartment is fairly consistently 10-20 degrees warmer inside than out), and good sleeping temperatures seem to be in the 60-68 degree range.

 

10 day forecast for Bellevue, WA

Now the weather where I’m moving to –

 

10 day forecast for San Bruno, CA

 

A little bit more rain in the forecast up here, but not much. In my 11 years of living in Bellevue it seems it hardly rains here at all, it’s as if the rain went around the city (it rains a lot up in the northern Puget Sound region known as the convergence zone).

But as far as people up here complaining it’s cold, and not summer like.. I don’t know if the above weather is typical or not for where I am going, but given the close proximity to water on both sides, I am not surprised it stays cool.

Just a few miles to the south and further inland the high temperatures jump a full 10+ degrees, I almost moved there, but decided against it because of the ~1 hour commute in each direction.

I wouldn’t mind it being drier wherever I was, quite frequently we are above 85% humidity in the Seattle area, I don’t suppose that will improve at my new place.

July 18, 2011

What I’ll miss most about Seattle

Filed under: General — Tags: — Nate @ 10:46 am

[ I will of course miss all my friends more than anything else, this post is not about friends but rather places ]

I have been thinking about my move coming up – my last full day in the area is this Thursday July 21st. (originally was going to be Saturday but because of moving issues it had to be moved sooner).

I mentioned in my original post that I don’t like Seattle, I can’t think of a single part of Seattle that I like, I don’t like the one way streets, the lack of parking (which on a recent news report was the #1 complaint of tourists), the traffic, and really am not a fan of the culture in general, I don’t think as a driver anything gets me more frustrated than driving in downtown Seattle. I was there yesterday in fact to try to find something in particular, I ended up just coming back home in frustration, never bothered to go in any stores because parking was such a pain. Part of the culture in Seattle that I don’t like is they are increasingly anti-car. Which is a fine view to have – you just won’t catch me dead living there!. I do like the east side though (where I live). It is interesting because a lot of folks I know in Seattle are the opposite, they hate the east side but love Seattle.

So I tried to think of if there was anything I did like about Seattle. I came up with two places in Seattle that I do love, and will miss. I’m not sure if I’ll be able to find a close replacement for either down in the Bay Area though.

The first place, which is far and away first place, there is no competition.

Cowgirls Inc

If you haven’t been to Cowgirls you really should check it out, words cannot properly describe the Cowgirls experience on a Friday or Saturday night after 9 PM. Don’t bother going in at 5-6-7 it’s pretty much like any other bar. After 9 things change though. I usually get there at 9, and by about midnight I have a tough time standing up so it’s time to go (I missed out on their 5th Anniversary party which I happened to go on that particular night but could not stay an extra 30 minutes to see the show because well I was destroyed). – oh and you’ll be doing a lot of standing as they remove the bar stools at around 9:30-10pm.

On these nights it is often wall to wall people, I stick close to the bar and defend my position (you’ll understand why if you go). The words kick ass don’t do it justice. I took another friend there for the first time this past Friday and he was blown away too, totally exceeded his expectations. I took another guy a couple of months ago he moved to Seattle last year, and he seemed like he was in shock most of the time he absolutely loved it.

I worked across the street from this place (which is on the corner of 1st and King street in Pioneer Square), back when it opened in 2004, though I was a very different person back then. We used to do software deployments that would start at 10pm and typically end around 2 or 3AM.

My co-workers liked to go there (at what seemed to be around 6-7pm) to play pool and eat the peanuts because they have you throw the shells on the floor and my co-workers loved that. The place is often pretty dead that early so no action.

My Favorite Cowgirl (moved away in 2010)

Even though I only went a few times a year the staff knew me pretty well when I came in, so that was nice. I am terrible with names and although I heard some of their names over the years I never retained them. My favorite cowgirl (left) was awesome, she did the craziest things including dancing on my shoulders on several occasions, the heels hurt the first couple of times but it was good pain 🙂

She moved away from Seattle last year which made me sad.

A funny story about the company I worked at that was across the street from this place. The last boss I had at that company was a really heavy drinker, one night him, the VP and a bunch of other folks (not including me I wasn’t at the company at that point) went to Cowgirls and drank a bunch.

The VP dared the director (my former boss) to jump up on the bar and start dancing. He apparently did it, and as a result one of the bouncers got pulled him off, got him in a headlock and kicked him out. Speaking of bouncers (they are called regulators there), there are quite a few, I would say upwards of 10 at any given time.

Another funny story about Cowgirls and VMware of all things – a local VMware rep was coming to meet me at the last company I was at, and his boss told him the wrong address(I gave the right address). So he called me up right when he was supposed to be there and asked where I was and told him he went to the wrong place. Here is how the conversation went (this was almost a year ago so the words are not completely accurate) –

  1. Him: so where are you located then?
  2. Me: on the corner of 1st and King street
  3. Him: where is that near?
  4. Me: it’s in Pioneer square
  5. Him: hmm, any more tips you can give?
  6. Me: It’s near the stadiums
  7. Him: Still not completely sure
  8. Me: Do you know where Cowgirls Inc is?
  9. Him: OH YEAH! Cowgirls!
  10. Me: I’m right across the street from that
  11. Him: I’ll be there in 10 minutes!

The place is run really smooth despite the crowds they really have it nailed down. I’ve never seen anything get out of hand while all the times I’ve been there. The staff is very talented, friendly and skilled at churning out drinks at a rapid rate.

If you go and have a good time don’t be afraid to tip big. Drink wise it’s a very affordable bar, even drinking with a friend I don’t think my tab(before tip) has ever been above $100. My tips at Cowgirls are anywhere from 80% to 130%  (+/- I don’t try to do percentages just come up with some number based on the bill). I also tip in cash, I have had on a couple occasions companies reject my tips on my CC bill I guess because they thought it was too much, can’t reject cash though! They are worth every penny.

There is really no other bar I’d rather go to, at least from ones I have been to in the U.S.  – It’s not a place to go to if you want to have a casual conversation, because you’ll just be yelling at each other to hear each other. So depending on the situation – perhaps go to a quiet bar first and talk about whatever you want to talk about then go to Cowgirls and have some real fun.

I have only two complaints about the place – one they don’t keep their web site up to date (they had a pop up running that was more than a year old at one point). The second, which has been more of a drawback for out of town friends than me is that they aren’t open every night. They are open every Friday and Saturday, and some other days if there is a sporting event on. I can’t tell you how many times I’ve driven folks there on off nights only to find them closed.

I will miss the place greatly, I do plan to come back and visit. I’ve known several of the staff there for several years now so I should see some familiar faces if I am back in town in a few months to a year.

On to number two, it is a distant but solid second.

Pecos Pit

I was introduced to this place while I was working at that same company in Pioneer Square back in 2003. I don’t know how famous it might be but it is the only place I will order a pulled pork bbq sandwich from. I’m actually going there today at noon to meet some friends.

I don’t believe I had ever had pulled pork until I had it at Pecos Pit, I’m not even sure if I had even heard of pulled pork until Pecos.

 

Pecos Pit is located on 1st ave, about a mile south of the stadiums in Seattle. They are open Monday – Friday only as far as I know and hours are something like 11-3PM. Outdoor seating only (or take out). Parking can be limited at times, oh and it’s cash only too (there is an ATM across the street in a pinch though I’ve never used it).

Probably the main reason I love Pecos is the sauce & spice. My standard order (which some of the staff know me so well that I don’t even have to say it) is Pecos Pork, Hot, Spike & Beans. Yes I order the hottest thing on their menu, few people do but I have been having it for so long I got used to it a long time ago. I started out with medium way back when, but at some point it didn’t seem hot enough (I think they adjusted their recipe to make things less spicy but not sure). I switched to hot, and while it really is hot, for me at least it’s by no means too hot. Of all my friends that I have taken there or met there, I think maybe only one or two others have gotten hot, most usually seem to get mild(I know of at least one that complains that mild is even too hot).

I live in Bellevue, very close to Dixie’s BBQ which is much more famous in the area because of the man. While the man passed away a year or two ago they still have the man sauce, which really is the hottest thing I have ever had. I enjoy the heat it gives but I really do not enjoy the flavor. I also don’t generally enjoy the flavor of the pulled pork at Dixie’s either. I’d much, MUCH rather drive to Seattle and get Pecos over Dixie’s. The only reason I would go to Dixie’s myself is to get a jar of the man sauce to use at home, they sell, what I think is 2 ounce jars of the sauce for something like $10. While I haven’t used it at home in many years, the time when I did, one jar of that stuff literally lasted me a year. I would apply it with toothpicks to meat to get the heat inside the meat and cook it. Really was good (and very hot). The man sauce I would have to say is probably 3-4-5x+ hotter than what the hottest is at Pecos.

I’ve had pulled pork at a couple other places as well but for me, nothing compares to Pecos (I’m sure real bbq from down south or east or whatever is as good or better but I haven’t been able to try any of that). So for the most part when I see pulled pork on a menu I don’t bother ordering it, unless I’m at Pecos.

Pecos is not a place to go if your not a meat eater, their menu is limited to pork, beef, and beans for the most part(which have meat in them). I’ve never tried the beef, never felt the need. I have heard that sometimes they run out of pork if the days are really busy but I haven’t come across that myself. Lines can be long on good weather days so be prepared to wait.

Well there you have it – there are more places I will miss from up here, but they aren’t unique and I’ll be able to find replacements for them pretty easily down in the Bay Area.

These places will be harder. I know there is a bunch of other bars that are sort of like Cowgirls around the country, I haven’t been to any myself, one of my friends who travels a bunch does, and told me at least of the ones he’s been to, nothing compares to Cowgirls Inc.

July 12, 2011

VMware jacks up prices too

Filed under: Virtualization — Tags: , — Nate @ 4:34 pm

Not exactly hot on the heels of Red Hat’s 260% price increase, VMware has done something similar with the introduction of vSphere 5 which is due later this year.

The good: They seem to have eliminated the # of core/socket limit for each of the versions, and have raised the limit of vCPUs per guest to 8 from 4 on the low end, and to 32 from 8 on the high end.

The bad: They have tied licensing to the amount of memory on the server. Each CPU license is granted a set amount of memory it can address.

The ugly: The amount of memory addressable per CPU license is really low.

Example 1 – 4x[8-12] core CPUs with 512GB memory

  • vSphere 4 cost with Enterprise Plus w/o support (list pricing)  = ~$12,800
  • vSphere 5 cost with Enterprise Plus w/o support (list pricing)  = ~$38,445
  • vSphere 5 cost with Enterprise w/o support (list pricing)         = ~$46,000
  • vSphere 5 cost with Standard w/o support (list pricing)           = ~$21,890

So you pay almost double for the low end version of vSphere 5 vs the highest end version of vSphere 4.

Yes you read that right, vSphere 5 Enterprise costs more than Enterprise Plus in this example.

Example 2 – 8×10 core CPUs with 1024GB memory

  • vSphere 4 cost with Enterprise Plus w/o support (list pricing) = ~$25,600
  • vSphere 5 cost with Enterprise Plus w/o support (list pricing) = ~$76,890

It really is an unfortunate situation, while it is quite common to charge per CPU socket, or in some cases per CPU core, I have not heard of a licensing scheme that charged for the memory.

I have been saying that I would expect to be using VMware vSphere myself until the 2012 time frame at which point I hope KVM is mature enough to be a suitable replacement (I realize there are some folks out there using KVM now it’s just not mature enough for my own personal taste).

The good news, if you can call it that, is as far as I can tell you can still buy vSphere 4 licenses, and you can even convert vSphere 5 licenses to vSphere 4 (or 3). Hopefully VMware will keep the vSphere 4 license costs around for the life of (vSphere 4) product, which would take customers to roughly 2015.

I have not seen much info about what is new in vSphere 5, for the most part all I see are scalability enhancements for the ultra high end (e.g. 36Gbit/s network throughput, 1 million IOPS, supporting more vCPUs per VM – number of customers that need that I can probably count on 1 hand). With vSphere 4 there was many good technological improvements that made it compelling for pretty much any customer to upgrade (unless you were using RDM with SAN snapshots), I don’t see the same in vSphere 5 (at least at the core hypervisor level). My own personal favorites for vSphere 4 enhancements over 3 were – ESXi boot from SAN, Round Robin MPIO, and the significant improvements in the base hypervisor code itself.

I can’t think of a whole lot of things I would want to see in vSphere 5 that aren’t already in vSphere 4, my needs are somewhat limited though. Most of the features in vSphere 4 are nice to have though for my own needs are not requirements. For the most part I’d be happy on vSphere standard edition (with vMotion which was added to the licensed list for Standard edition about a year ago) the only reason I go for higher end versions is because of license limitations on hardware. The base hypervisor has to be solid as a rock though.

In my humble opinion, the memory limits should look more like

  • Standard = 48GB (Currently 24GB)
  • Enterprise = 96GB (Currently 32GB)
  • Enterprise Plus = 128GB (Currently 48GB)

It just seems wrong to have to load 22 CPU licenses of vSphere on a host with 8 CPUs and 1TB of memory.

I remember upgrading from ESX 3.5 to 4.0, it was so nice to see that it was a free upgrade for those with current support contracts.

I have been a very happy, loyal and satisfied user & customer of VMware’s products since 1999, put simply they have created some of the most robust software I have ever used (second perhaps to Oracle). Maybe I have just been lucky over the years but the number of real problems (e.g. caused downtime) I have had with their products has been tiny, I don’t think it’s enough to need more than one hand to count. I have never once had a ESX or GSX server crash for example. I see mentions of the PSOD that ESX belches out on occasion but I have yet to see it in person myself.

I’ve really been impressed by the quality and performance (even going back as far as my first e-commerce launch on VMware GSX 3.0 in 2004 we did more transactions the first day than we were expecting for the entire first month), so I’m happy to admit I have become loyal to them over the years(for good reason IMO). Pricing moves like this though are very painful, and it will be difficult to break that addiction.

This also probably means if you want to use the upcoming Opteron 6200 16-core cpus (also due in Q3) on vSphere you probably have to use vSphere 5, since 4 is restricted to 12-cores per socket (though would be interesting to see what would happen if you tried).

If I’m wrong about this math please let me know, I am going by what I read here.

Microsoft’s gonna have a field day with these changes.

And people say there’s no inflation going on out there..

sigh

Netflix jacks up rates – I cancelled

Filed under: Random Thought — Tags: — Nate @ 2:58 pm

All that trouble tracking down why my Netflix HD streaming was not working for nothing? I guess so. Netflix sent me an email a short time ago said they were going to increase the cost of my plan from $10 to $16. So I closed my account. They raised the price by a buck from $9 to $10 last November.

I normally wouldn’t mind the increase in charges if I was using the service, but I checked my email archives I’ve had the same DVD sitting waiting to be played since May 31st, and the last time I streamed a “full” movie or tv show from their streaming service looks to be January 2010 based on the “How was the quality of X?” emails. I didn’t think it was that long ago. I have streamed short segments of a bunch of stuff over the past year but always got bored of what I was watching so never watched more than a few minutes at a time.

If they had a better selection …..especially on the streaming side, I swear every time I’ve gone there in the past 6 months I have not noticed a single thing I wanted to stream. I suppose part of that is having a Tivo for so long I really don’t keep track of what kind of things come out, frequently coming across TV shows for the first time long after they had been canceled.

I have a week to return this DVD that has been sitting here for almost 2 months, I guess I will go pop it in the mail because I likely won’t get around to watching it in the next week.

What would of been nicer of course is if Netflix was better at being able to bill based on actual usage, if so my bill probably should of been $0.99/mo 🙂

Netflix’s content costs are apparently about to skyrocket so they need to get ready for that by raising rates..

[..] Barclays analyst Douglas Anmuth: He figures Netflix will have a total streaming commitment of $2 billion by the end of 2011.

Let me know when we have a video streaming service that is fulfills the dream of this Qwest commercial. I’m not holding my breath.

I’m more than happy to pay for premium services or products, in this case I was just paying them for the convenience that I might use it. I’ve rented 11 DVDs (10 of which I have watched) so far in 2011 through Netflix, and 17 in 2010.

« Newer Posts — Older Posts »

Powered by WordPress