Dec/1110
VMware increases core counts in 4.1 licensing
TechOps Guy: Nate
I just came across this mention on AMD’s blog. They note that vSphere 4.1 Update 2 included a CPU licensing change -
For the AMD Opteron 6200 and 4200 series (Family 15h) processors, ESX/ESXi 4.1 Update 2 treats each core within a compute unit as an independent core, except while applying licenses. For the purpose of licensing, ESX/ESXi treats each compute unit as a core. For example, a processor with 8 compute units can provide the processor equivalent of 16 cores on ESX/ESXi 4.1 Update 2. However, ESX/ESXi 4.1 Update 2 only requires an 8 core license for each 16-core processor.
I had not heard of that before, so it’s news to me! So not only is the physical cost of the Opteron 6200 cheaper than the 6100, the licensing cost is half as much (per core). AMD’s blog post above shows some pretty impressive results where a pair of quad socket 6200 blades outperforming a pair of quad socket 10-core Intel blades(2 sockets populated per blade) and at the same time the 6200 solution costs half as much (per VM). Though it’s also comparing vSphere 4.1 vs 5.0, since the Opteron 6200 results seem to be the first vSphere 5.0 VMmark results posted. Also the Intel solution has twice the ram as the Opteron but still loses out.
Based on what I see it seems VMmark is more CPU bound than memory(capacity bound), which I suppose I can understand but still in the vast majority of situations the systems are not CPU bound. People tend to load up more on CPUs so they can get more memory capacity. I won’t have real numbers for probably two months but I’m expecting CPU usage on this new cluster I am building to be at least half the amount of memory usage.
The change sounds Oracle-esque in licensing where they have fairly complicated decisions they made to determine how many “Oracle cores” you have on your physical processor.
I am traveling tonight to Atlanta to deploy a new vSphere cluster with Opteron 6100s, I was going to go with vSphere 5 because of the license limits on vSphere 4.1 not supporting 16 core processors. Now I see 4.1 does support it so I have about 48 hours to think about whether or not I want to change my mind. I do like vSphere 5′s inclusion of LLDP support, more vCPUs per VM. Though really even now after I have been looking through what is in vSphere 5 I don’t see anything game changing, nothing remotely, in my opinion like the change to vSphere 4.0 from ESX 3.5.
Weigh the benefits of what’s new in vSphere 5 vs having the ability to have unlimited memory(well, up to 1TB, which for me is unlimited from a practical standpoint) in my hosts for no additional licensing cost…
I’m already licensed for vSphere 5 since we bought it after the deadline of the end of September.
Mad props to AMD for getting VMware to tweak their licensing.
Decisions, decisions..
Jul/115
Wired or Wireless?
TechOps Guy: Nate
I’ll start out by saying I’ve never been a fan of Wifi, it’s always felt like a nice gimmick-like feature to have but other than that I usually steered clear. Wifi has been deployed at all companies I have worked at in the past 7-8 years though in all cases I was never responsible for that (I haven’t done internal IT since 2002, at which time wifi was still in it’s early stages(assuming it was out at all yet? I don’t remember) and was not deployed widely at all – including at my company). I could probably count on one hand the number of public wifi networks I have used over the years, excluding hotels (of which there was probably ten).
In the early days it was mostly because of paranoia around security/encryption though over the past several years encryption has really picked up and helped that area a lot. There is still a little bit of fear in me that the encryption is not up to snuff, and I would prefer using a VPN on top of wifi to make it even more secure, only really then would I feel comfortable from a security standpoint of using wifi.
From a security standpoint I am less concerned about people intercepting my transmissions over wifi than I am about people breaking into my home network over wifi (which usually happens by intercepting transmissions – my point is more of the content of what I’m transferring, if it is important is always protected by SSL or SSH or in the case of communicating with my colo or cloud hosted server there is a OpenVPN SSL layer under that as well).
Many years ago, I want to say 2005-2006 time frame, there was quite a bit of hype around the Linksys WRT-54G wifi router, for being easy to replace the firmware with custom stuff and get more functionality out of it. So I ordered one at the time, put dd-wrt on it, which is a custom firmware that was talked a lot about back then (is there something better out there? I haven’t looked). I never ended up hooking it to my home network, just a crossover cable to my laptop to look at the features.
Then I put it back in it’s box and put it in storage.
Until earlier this week, when I decided to break it out again to play with in combination with my new HP Touchpad, which can only talk over Wifi.
My first few days with the Touchpad involved having it use my Sprint 3G/4G Mifi access point. As I mentioned earlier I don’t care about people seeing my wifi transmissions I care about protecting my home network. Since the Mifi is not even remotely related to my home network I had no problem using it for extended periods.
The problem with the Mifi, from my apartment is the performance. At best I can get 20% signal strength for 4G, and I can get maybe 80% signal strength for 3G, latency is quite bad in both cases, and throughput isn’t the best either, a lot of times it felt like I was on a 56k modem. Other times it was faster. For the most part I used 3G because it was more reliable for my location, however I do have a 5 gig data cap/month for 3G so considering I started using the Touchpad on the 1st of the month I got kind of concerned I may run into that playing with the new toy during the first month. I just checked Sprint’s site and I don’t see a way to see intra month data usage, only data usage for the month once it’s completed. The mifi tracks data usage while it is running but this data is not persisted across reboots, and I think it’s also reset if the mifi changes between 3G and 4G services. I have unlimited 4G data, but the signal strength where I’m at just isn’t strong enough.
I looked into the possibility of replacing my Mifi with newer technology, but after reading some customer reviews of the newer stuff it seemed unlikely I would get a significant improvement in performance at my location, enough to justify the cost of the upgrade at least so I decided against that for now.
So I broke out the WRT-54G access point and hooked it up. Installed the latest recommended version of firmware, configured the thing and hooked up the touchpad.
I knew there was a pretty high number of personal access points deployed near me, it was not uncommon to see more than 20 SSIDs being broadcast at any given time. So interference was going to be an issue. At one point my laptop showed me that 42 access points were broadcasting SSIDs. And that of course does not even count the ones that are not broadcasting, who knows how many there are there, I haven’t tried to get that number.
With my laptop and touchpad being located no more than 5 feet away from the AP, I had signal strengths of roughly 65-75%. To me that seemed really low given the proximity. I suspected significant interference was causing signal loss. Only when I put the touchpad within say 10 inches of the antenna from the AP did the signal strength go above 90%.
Looking into the large number of receive errors told me that those errors are caused almost entirely by interference.
So then I wanted to see what channels were most being used and try to use a channel that has less congestion, the AP defaulted to channel 6.
The last time I mucked with wifi on linux there seemed to be an endless stream of wireless scanning, cracking, hacking tools. Much to my shock and surprise these days most of those tools haven’t been maintained in 5-6-7-8+ years. There aren’t many left. Sadly enough the default Ubuntu wifi apps do not report channels they just report SSIDs. So I went on a quest to find a tool I could use. I finally came across something called wifi radar, which did the job more or less.
I counted about 25 broadcasting SSIDs using wifi radar, nearly half of them if I recall right were on channel 6. A bunch more on 11 and 1, the other two major channels. My WRT54G had channels going all the way up to 14. I recall reading several years ago about frequency restrictions in different places, but in any case I tried channel 14 (which is banned in the US). Wifi router said it was channel 14, but neither my laptop nor Touchpad would connect. I suspect since they flat out don’t support it. No big deal.
Then I went to channel 13. Laptop immediately connected, Touchpad did not. Channel 13 is banned in many areas, but is allowed in the U.S. if the power level is low.
Next I went to channel 12. Laptop immediately connected again, Touchpad did not. This time I got suspicious of the Touchpad. So I fired up my Palm Pre, which uses an older version of the same operating system. It saw my wifi router on channel 12 no problem. But the Touchpad remained unable to connect even if I manually input the SSID. Channel 12 is also allowed in the U.S. if the power level is low enough.
So I ended up on channel 11. Everything could see everything at that point. I enabled WPA2 encryption, enabled MAC address filtering (yes I know you can spoof MACs pretty easily on wifi, but at the same time I have only 2 devices I’ll ever connect so blah). I don’t have a functional VPN yet mainly because I don’t have a way (yet) to access VPN on the Touchpad, it has built in support for two types of Cisco VPNs but that’s it. I installed OpenVPN on it but I have no way to launch it on demand without being connected to the USB terminal. I suppose I could just leave it running and in theory it should automatically connect when it finds a network but I haven’t tried that.
So on to my last point on wifi – interference. As I mentioned earlier signal quality was not good even being a few feet away from the access point. I decided to try out speedtest.net to run a basic throughput test on both the Touchpad and the Laptop. All tests were using the same Comcast consumer broadband connection
| Device | Connectivity Type | Latency | Download Performance | Upload Performance |
|---|---|---|---|---|
| HP Touchpad | 802.11g Wireless | 18 milliseconds | 5.32 Megabits | 4.78 Megabits |
| Toshiba dual core Laptop with Ubuntu 10.04 and Firefox 3.6 | 802.11g Wireless | 13 milliseconds | 9.46 Megabits | 4.89 Megabits |
| Toshiba dual core Laptop with Ubuntu 10.04 and Firefox 3.6 | 1 Gigabit ethernet | 9 milliseconds | 27.48 Megabits | 5.09 Megabits |
The test runs in flash, and as you can see of course the Touchpad’s browser (or flash) is not nearly as fast as the laptop, not too unexpected.
Comparing LAN transfer speeds was even more of a joke of course, I didn’t bother involving the Touchpad in this test just the laptop. I used iperf to test throughput(no special options just default settings).
- Wireless – 7.02 Megabits/second (3.189 milliseconds latency)
- Wired – 930 Megabits/second (0.3 milliseconds latency)
What honestly surprised me though was over the WAN, how much slower wifi was on the laptop vs wired connection, it’s almost 1/3rd the performance on the same laptop/browser. I justed measured to be sure – my laptop’s screen (where I believe the antenna is at) is 52 inches from the WRT54G router.
It’s “fast enough” for the Touchpad’s casual browsing, but certainly wouldn’t want to run my home network on it, defeats the purpose of paying for the faster connectivity.
I don’t know how typical these results out there. One place I recently worked at was plagued with wireless problems, performance was soo terrible and unreliable. They upgraded the network and I wasn’t able to maintain a connection for more than two minutes which sucks for SSH. To make matters worse the vast majority of their LAN was in fact wireless, there was very little cable infrastructure in the office. Smart people hooked up switches and stuff for their own tables which made things more usable, though still a far cry from optimal.
In a world where we are getting even more dense populations and technology continues to penetrate driving more deployments of wifi, I suspect interference problems will only get worse.
I’m sure it’s great if the only APs within range are your own, if you live or work at a place that is big enough. But small/medium businesses frequently won’t be so lucky, and if you live in a condo or apartment like me, ouch…
My AP is not capable of operating in the 5Ghz range 802.11a/n, that very well could be significantly less congested. I don’t know if it is accurate or not but wifi radar claims every AP within range of my laptop(47 at the moment) is 802.11g (same as me). My laptop’s specs say it supports 802.11b/g/n, so I’d expect if anyone around me was using N then wifi radar would pick it up, assuming the data being reported by wifi radar is accurate.
Since I am moving in about two weeks I’ll wait till I’m at my new apartment before I think more about the possibility of going to a 802.11n capable device for reduced interference. On that note does any of my 3-4 readers have AP suggestions?
Hopefully my new place will get better 4G wireless coverage as well, I already checked the coverage maps and there are two towers within one mile of me, so it all depends on the apartment itself, how much interference is caused by the building and stuff around it.
I’m happy I have stuck with ethernet for as long as I have at my home, and will continue to use ethernet at home and at work wherever possible.
Nov/100
Busy busy
TechOps Guy: Scott
I know many in IT are so busy that even taking a moment to reach down into your sock just to scratch an itchy ankle will likely disappoint some individual, group, project, initiative, deadline, milestone, sprint, release, vision, journey, senior mgmt, customer expectation, etc but the last two weeks have been ridiculous for me. It’s been all work and no play even with decent off shore winds today and some steady +4 footers (yeah if you don’t understand the reference then you don’t know what you’re missing). That’s saying something.. I normally don’t miss a good session and when the waves are really working I am normally not 
Anyhow, I want to keep this short but I have been very busy working with some cool tools I’d like to pass along (nothing new here but if you aren’t familiar then check it out):
- puppet (2.6.1 release I have SRPM’s if interested)
- mcollective
- icinga
- stashboard (I’d like to get this to run sans googleappengine)
- this (SRPM’s as well… interesting benchmarks )
- git (of course)
I won’t comment on all of the listed items but I must say that stashboard is really freaking cool and I would love to have a fork that can be run behind the firewall. The restful API’s for doing just about everything is appealing and it means I won’t have to come up with my own simplified NOC dashboard for end-users. Also, I am really liking icinga (nagios fork) but now that zenoss is giving away a free esxtop zenpack I’m not sure which direction to go for my monitoring / alerting / trending NOC software.
Hope you enjoy the run down and thanks for reading.
Oct/100
Amazon freebies
TechOps Guy: Scott
Two part post. Firstly, I would like to brief the community of a TechOpsGuys Public Service Announcement and inform you (if you haven’t heard) that Amazon is offering free cloud services starting November 1. I know, old news (two days) but the James Hamilton blog has an excellent reader’s digest version of the announcement here. Yes, lol his picture.. the guy is a damn rockstar with that dew but I encourage you to follow his blog and more formal literature which is often released after major industry expos. You will thank me after spending a few hours looking over his brilliant body of work.
Even more Amazon details can be found here.
Second, I will definitely be playing with each and every services API using some of the excellent projects over at the githubs. If the frameworks around Amazon aren’t drop dead simple or elegantly written I’ll probably write my own and post it to github so stay tuned if you’re interested. Of interest to me initially is testing some automated app deployment tools like chef, fabric, vlad, and some creative special sauce just for fun. I typically only write backend data integration type stuff since I function as a sysadmin all day but I’m excited about writing custom apps (maybe even frontend) and exercising some creativity. I’ll be sure to keep you all informed and in the meantime check out the tools mentioned above and comment about some that you prefer.
One last thing, if this topic interests you head on over to the devops toolchain and poke around for awhile. Maybe even signup for the mail list and share some common interests with like minded individuals. I subscribed to the digest and enjoy its light reading.
Oct/100
Inhale… PKI and XMLRPC… exhale
TechOps Guy: Scott
I’ve been working a project the last few days to automate the handling of creation, revocation, and all around management of a PKI. Look if I want to keep a team of two guys in charge of hundreds and even thousands of engineers, multiple sites, terabytes of data, private cloud, and continued explosive growth then automation is key.
Anyhow, I’m finally finished and very pleased with the end result. I believe it to be a fair balance between security, functionality, and extensibility. Without giving away all of the keys to the kingdom the architecture is as follows (for creation):
- Account provisioning automation tier submits authenticated XMLRPC request to the isolated CA VM over SSL with the common name as part of the message.
- CA creates the key and encrypts it with a random password, generates the CSR, signs the CSR, generates the encrypted PKCS12 file, and publishes the certs to the applicable LDAP object attributes.
- XMLRPC instance responds back with (N)ACK
- XMLRPC instance logs the entire lifecycle of key management consistently
The corollary of course (revocation):
- Account provisioning automation tier submits authenticated XMLRPC request to the isolated CA VM over SSL with the common name as part of the message.
- CA revokes the certificate subsequently updating the CRL
- The CRL gets pushed to a git repository with all of the other configs; published with configuration management utilities (think cfengine, puppet, or chef)
- The CRL finally makes its way to the systems that matter via said config management software and dependent services are bounced.
What’s important to note is the fact that I use a very minimal XMLRPC interface to the PKI. No administrators logging in and rooting around doing manual one offs with the PKI. Every interaction is now forced to be consistent, audited, and in my opinion secure. Even better the certificates are signed according to the purpose of creation (obviously following RFC3280 as closely as possible). More specifically, if a certificate is for a user then the proper extension attributes are populated and likewise for server-side components. This further ensures compliance with the PKI and TLS related standards.
As far as logging, I made sure to keep logging very consistent with the rest of the framework that the PKI component is now just another plugin of. Basically, this means a tuple of (boolean, ‘msg’) which allows for clean and easy flow control. An example is:
c = ssl.PKI()
log('Adding new user to PKI')
rc, msg = c.NewUser(params)
if rc:
log('Successfully added new user to PKI')
<do something>
else:
return (rc, msg)
That code snippet is 100% made up on the spot but you see that having consistent logging and return codes is quite valuable and frankly easy to read.
Ok, so that’s cool but I would be lying if I said I got it right the first time. This project went through a few different POC’s including reliance upon M2Crypto, python-openssl, a mix of the two, libPKI (C), some mixed up crap out of github, and even a wrapper around the openssl command line utility. The cat’s not out but suffice to say I ultimately went with what I thought was the most most feature rich and elegant solution.
Such work coming from the IT group is what I think of when I read about Next Practice or with a more modern spin Infrastructure as Code. It’s what I expect out of a certain elite few on my team and if your a big enterprise without such individuals then you either don’t know what you are missing or are actively trying to poach the individual out of some other nervous bastards clinched grip. The individual is for damn sure employed and you better have a nice package or some exciting work like self-driving cars to steal her.
Oct/101
Open Source clouds
TechOps Guy: Scott
Not sure how I missed ElasticHosts (maybe because they are UK only right now) but they’re a seemingly viable public cloud based on KVM. Too cool and the only one of its kind AFAIK (please comment if you are aware of others). KVM is an awesome project and I’ve been following the development mailing list for quite a while so I’d like to think I know just how cool it is. For those of you who claim it isn’t “enterprise class” or even the more ridiculous claim that it “… isn’t a type 1 hypervisor” get a life and learn what a kernel module is. The HCL is far more superior to any other hypervisor not to mention the code contributions are coming from quite highly respected engineers from a number of different companies and ISV’s.
Still, the technology mentioned serves as merely a tangent in comparison to this weeks news and releases in the Open Source cloud space. The folks over at OpenStack pushed out a development release of the “compute” variety and production release of the “storage” tier. Fantastic indeed but I am more excited about the OpenNebula project and the major release just announced. The feature set of OpenNebula is unbelievable and growing (see the link). I currently manage a vSphere cluster and when comparing the vCloud Director to some of these Open Source alternatives I’m leaning (nay running) toward the more freely available, extensible, and easier to automate solution for a private cloud deployment. Sure if all you know how to do is click Next -> Next -> Finish then OpenNebula and the like will scare you but if you really want to unlock the feature sets then vCloud Director (Oracle, .Net, and 64 bit windows requirements) feels more like IE6 in today’s browser wars.
Oct/100
Runt post
TechOps Guy: Scott
I always liked how the guys over at the PacketPushers podcast describe the short podcasts as “Runts”. If you’re a networking junkie the reference is obvious. Having said that, this is a a short and unexpected post on something that just came across my Google reader.
If you have been following the industry buzz around DevOps, Agile software development, and continuous deployment then you must respect the guys over at Wealthfront (formally kaChing). No need for this post to be any longer, just read this. Wow.
Oct/100
Sunday reading
TechOps Guy: Scott
I was enjoying a beverage and reading an interesting LWN.net article on the topic of the Linux block layer and SSD’s. Most interesting to me were the references to lessons learned in the networking stack. Quite rightly in my opinion. If you’ve been a Linux admin with an eye on networking since about the 2.6.9 release (guessing but sounds about right) then you are already familiar with technologies like:
Obviously, these were needed to keep up with increasing rate of PPS and throughput requirements of +1Gb. Hell, one of the most advertised features of NAPI is polling under pressure instead of 100% interrupt driven. These enhancements are now being dusted off and reviewed once more but for inclusion in a completely different subsystem.
The article was illuminating to me since I had not been aware of the technical issues in the block layer. I wasn’t blind-sided by the news since it’s obvious to any technologist that a jump from a few hundred IOPS per device to several tens of thousands or hundreds of thousands is going to identify inefficiencies; not to mention the block subsystem has been designed to avoid drive seeks which is largely overhead when dealing with SSD’s. It’s seems the bottleneck is to remove locking in many locations, better handle SMP systems, and figure out how in the world to best handle crappy controllers. For instance, newer NIC’s come with multiple RX/TX queues so efficient use of SMP systems can be hardware offloaded (especially when the address / port tuple is hashed in hardware) but AFAIK this is not the case in storage controllers. It will be interesting to see how much the improvements to the block layer mirror those of the networking subsystem.
I’m still getting used to blogging and can’t quite tell if these posts are too long or too short so hang with me as I find my pace . I’ll have another one here shortly about how cool the Mantis bugtracker is and how I extended the built-in SOAP API and made use of it to automate interactions. Very exciting.
Oct/100
Open Source Innovation
TechOps Guy: Scott
I’ve been doing a bit of window shopping in preparation for a private cloud implementation next year and am very excited about the maturity, speed, and awesomeness of the open source community. Awesomeness of course means API’s and python / ruby bindings for a number of these projects or in the case of the openstack nova project it’s largely written in python. Many of the technologies used with openstack ( rabbitmq, nginx, and redis) scale quite nicely and are used in production at some very large infrastructures for which an IT guy can find some very cool tech talks on. These tools and crew are the way of the future and IT should really get familiar with them and comfortable extending with some special sauce which unlocks the true value-add in your vertical.
Technologies like ceph and sheepdog look equally cool on the file system side of the house. Although still pretty green, this video highlights some of the features of sheepdog which could become quite cool indeed. 4Mb chunks distributed across however many commodity servers sans meta data server / bottleneck. Now, don’t be silly and think I’m serious about putting something like sheepdog or ceph in production today but the technology is moving fast and it’s absolutely worth following.. especially since ceph was included upstream in 2.6.34 and sheepdog is currently usable by qemu (which implies KVM). Not to mention, sheepdog in documentation appears to be incredibly simple to understand, manage, and operate. Quite a contrast to its competition. I’ve been a victim of Redhat’s GFS in the past and toyed with other distributed file systems like VMFS (before going directly to NFS and collecting $200) and have decided that they are far too complicated and awful for me.
Simplicity is the name of the game now for an infrastructure guy like myself. I’m done hyper analyzing every minute detail of the stack and want to accomplish the task at hand with significantly less ramp up time, cost, and complexity. I need a licensing model that scales (GPL, BSD, Apache, etc) and a product that is extensible. Your engineering and finance department will thank you!
Jun/101
8 Things Startups Must Know Before Signing a Lease
TechOps Guy: Jason
I’ve had the opportunity to perform office selection and relocation a number of times in my career and I wanted to give the tech community some quick and dirty lessons when considering new office space.
#1 – Direct vs. Sublease
Before committing your company to 5+ year lease you might want to consider subleasing from an existing company that is downsizing. In this market cash conscious startups can easily negotiate short and long-term subleases. Remember shorter is better and allows for greater flexibility and potentially an easier exit should you become acquired.
#2 – Hire a Tenant Broker, Not a Commercial Real Estate Agent
I know that everyone has a friend that is in the business, but remember you need an advocate. Tenant Brokers do not offer their services to Building Owners so they are free to aggressively negotiate on your behalf.
#3 – Whenever Possible Create Pocket Space
Creating a pocket space allows you to grow into your new office without paying for the full office footprint up front. Can you set aside 500/1000/2000 sq ft for future growth? If so, add it to the lease.
#4 – Controlling Costs During Build Out
Assign a high performing individual to manage vendors and change requests. Make this person responsible for managing the day-to-day decisions that arise as you complete your build out. Think about it, every little change request can add up to a bunch of dollars quickly.
#5 – Do Not Create Specialized Offices or Spaces for Individual Contributors
The fact is your workforce changes over time, your office space generally does not unless you are willing to cough up money for tenant improvements as your needs change. Keep your office flexible and the ability to scale as you add more employees.
#6 – Termination Clauses
Planning on being acquired before your lease runs out? Are you afraid you will grow out of the space the building provided? See if you can be given the first right of refusal on adjacent space, or spaces within the building. If they cannot accommodate your growth then add that to the termination clause.
#7 – Commuting & Service Friendly
Is this location commuter friendly? Can your employees easily access basic services including great coffee shops and lunch places? Does your space have a kitchen and fridge? Your talent does care where you are located and basic amenities.
#8 – Employee Safety
As the employer you need to think carefully about a building’s location and parking facilities and their safety. Not everyone is the same, but consider what it might be like for one of your employees to be leaving the office at 9pm – will your employees feel safe?
Follow Nate
