TechOpsGuys.com Diggin' technology every day

30Jan/14Off

From WebOS to Android: first 30 days

TechOps Guy: Nate

So as all 9 readers of my blog know I have been a long time WebOS user. Really it was my first real smart phone back in 2009 the Palm Pre. The first and only Palm branded product I have ever owned (other than Pre accessories - my next WebOS device was post HP acquisition).

RIP WebOS

Anyway as I have written about in the past, for a while after HP killed the hardware I was holding out some degree of hope that the software would find a new home, obviously that hope dwindled as time went on and as of about probably 9-10 months ago I decided to kill off whatever hope that was left in me. The current state of WebOS is quite poor, I felt even while HP owned WebOS - every day that went by it was falling further and further behind, they had some unique technology advantages that still shine today but that wasn't nearly enough to make up for the shortfalls. HP later sold the WebOS hardware group to LG to make smart TVs (which seemed to debut at CES this month), and more recently HP sold the remaining patents that they had involving Palm and WebOS to Qualcomm.

Honestly it was somewhat depressing to see the die hard WebOS fans say on what is probably one of the very few WebOS community sites left. Some held really high hopes of what was to come. It didn't(and doesn't make sense to me). The maintainers of the site even stopped posting news articles more than six months ago because there was just nothing to write about (and the six months prior the articles were really scraping the bottom of the barrel for content).

Deciding to jump ship

Around the middle of last year I was getting tired of the software glitches in WebOS that I have endured over the years, knowing they will never be fixed, and Open WebOS is even today little more than a pipe dream (from the comments I've read I'd wager it's at least 2-3 years away from anything usable as a phone and by then it will have even more catching up to do, so really it seems to be a waste of time for anything other than tinkering). I thought about it off and on and decided that the likely candidate replacement was going to be the Samsung Galaxy Note 3, whenever it was going to be released.

Factors going into my decision were I wanted it to be fast, have plenty of storage, have a big enough screen so my big fingers could type on it, and decent battery life. I also wanted it to be Linux friendly as I use Linux on my laptop and desktops.   The specs of the Note 3 weren't released at the time so I decided to wait to see what else came about just in case I think I would want something different. Finally the Note 3 was announced and released and had strong reviews across the board.

I saw a bunch of other devices but none of them stood out to me more than the Note 3.

Keep in mind I have never used an Android or IOS device for more than say 5 minutes so my knowledge of either was extremely limited.  One thing I did like about the Note 3 was it's support for 64GB of internal flash in addition to 64GB of MicroSD expansion. So I decided to wait until I could get a 64GB Note3 to have 128GB of local storage, that would be pretty nice. Searching is annoying because so many results come from people mentioning the Note 3 with 64GB of microSD storage..

So I waited, and waited. Looked around a lot, plenty of news sites reporting 64GB was supported but could not find a sign of anyone -- not even one person in the world -- saying that they had it or knew where to buy it. Even now, doing a very casual search I do not see anyone with a 64GB Note 3.

So December 22nd comes around and I'm at a bar watching a football game, and thinking about going to Best buy across the street to buy it after the game as they were offering it at $199 which is $100 less than anyone else obvious that I saw, and I could walk away with it that day.

So I went and bought the 32GB version, with a 64GB Micro SD card.

First impressions

It's a big phone for sure, the Pre3 has a 3.58" screen and the Note 3 has a 5.7" screen. The Pre3 is a slider phone with a real keyboard so that adds extra heft. In fact the Note 3 is only 13 grams heavier than the Pre3 - a difference I can't even feel.

Obviously the Pre3 is outclassed in every way:

  • I have six times more storage(16GB vs 96GB)
  • I have six times more memory(512MB vs 3GB)
  • I have quad core 2.3Ghz vs single core 1.4Ghz
  • GPU I'm sure is significantly better
  • I have 1.7 million more pixels on the screen (800x480 vs 1920x1080)
  • I have full LTE support (AFAIK no WebOS device had LTE) - and hey - I'm already paying  an extra $40 or $50/mo for 5GB of data with a Mifi data plan, so might as well leverage LTE right?
  • Significantly better camera (and camera fuctions)
  • I can actually use Bluetooth and 2.4Ghz wifi at the same time (could not do that on the Pre3, would get massive interference on Bluetooth)
  • Much bigger battery and I believe much better battery life
  • I can have tons of photos without the OS crapping out (several hundred supported in WebOS, so far I have more than 12,000 on my Note 3 and I got plenty of room to grow I think)

I could go on...

Anyway, from an overall user experience perspective I have found making the adjustment from WebOS to Android much easier than I had originally expected.  I do like having a plethora of options to play with, that is something WebOS did not have (though out of the box WebOS had a good user experience other than being slow).

Thirty days or so into my purchase there are really only three things I miss from my WebOS days:

  • Wireless charging (this is a huge one for me, I had been using wireless charging for the previous four years -- I know Note 3 has wireless charging support so I will have that soon)
  • Unable to quickly silence notification alarm. Working in operations my phone acts as a pager. I have a very loud, long, and annoying notification message for alerts. The first time that noise went off waking me up at 2AM I about had a heart attack(click the link to listen to it). With my WebOS phones I could just hit the power button and the sound would mute immediately. Not so on this Note 3. I have looked online and this not an uncommon complaint about Android (though some device manufacturers offer this ability). I have seen people requesting this feature going back at least three years. This is quite annoying to not have....

Speaking of which the placement of the power button exactly opposite to that of the volume rocker is not good in my opinion, I find myself pressing the volume button on accident just to press the power button(which I think causes problems for trying to take a screen shot more details on that below). On the WebOS phones the power button is on the top.

  • The Note 3 is not smart enough to determine where to put a phone call. On WebOS for example if I have a bluetooth headset paired with the phone, and a call comes in -- and I answer the call with the phone (not the headset) the call is placed on the phone. On the Note 3 (also noticed this on my last "feature" phone) if a headset is paired(and connected) the call always goes to the headset. I've had several occasions where people have hung up on me with me saying hello???? not realizing that the call had been sent to the headset. So I have to answer the call, and wait a second(to see if the headset is paired, since they auto pair when in range often times) then hit the headset button to transfer the call back to the phone if I am not in immediate reaching range of one of my many bluetooth headsets. That process takes a good 3-5 seconds where the caller is left in limbo.

None of them are deal breakers of course, overall the experience is positive, and I'm glad I made the switch. I could go on for quite a while with the issues I have had with WebOS over the years but that's all in the past now. I still use my WebOS tablets, though these days the browser is so old and decrepit that I really only use them for about 4 different web sites(in all cases I disable javascript to get passable performance). They do still make great digital picture frames (as long as you have less than say 1,000 images). They also are good video playback devices with good audio (though the headset volume is really low, too low to use on an airplane to watch video).

On the Note 3 I really like the stylus (or S-Pen as they call it). I use it tons of times throughout the day. It's really good for precision. It's also the only way I've been able to take a screen shot in Android. I've found a few websites that have upwards of a half dozen ways to take a screen shot and none of them work for me(I think my timing in pressing the buttons is not perfect, but it shouldn't have to be).  But the S-pen has a function that I just click on and it works every time.  The S-pen has a bunch of other functions that for the most part I haven't used yet.

The camera is quite good as well it has so many features (the Pre3 camera had literally one feature - the flash - on/off/auto). I took a couple panoramic shots on my recent holiday road trip. One thing I liked about the Pre3 camera was it was fast. You press the button and instantly you have a picture - the Note 3 at least in auto mode (again haven't messed with it much) you press the button and it tries to focus and then take the picture. You can do burst mode and take tons of pictures (whereas with Pre3 you have to keep hitting the button but it is fast! - though focus isn't always right).

Battery life isn't quite as good as I was expecting given the rave reviews I have seen since the Note 3 was released. It can be confusing, I could watch a 45 minute video and the battery will drop 4-5%, or I could play a game for 10-15 minutes and the battery drops 8-10%. I have been so used to wireless charging and just having my phone charge constantly I find myself plugging and unplugging my Note 3 a half dozen or more times a day just to keep the battery up(I'm obviously worried about the durability of the micro USB connector). I haven't had it drop much below 50%. I'm sure it could go a full day with typical use, but I just don't like seeing it below 70-80% if I'm close to a charger.

My Pre3 on a regular day probably spent 60% or more of the day/night sitting on a charger. The Note 3 will do the same once I get wireless charging hooked up. Though it's going to cost a bit of $ - maybe $250 or so to get enough good charging stations and the charging backplate. Sort of surprised the price of wireless charging hasn't really moved much in the past four years..

I don't have any protective cover or case on the phone. I don't plan to get any, I treat my electronics with a good amount of care.

I do miss the USB drive mode of the WebOS devices though, just plug it in to any computer and it turns into a USB drive (though all phone functions are off during this). With the Note 3 it uses that strange media standard and at least at the moment I can only connect it to a windows computer to copy files onto it (and it doesn't get a drive letter either). It works fine from within VMware workstation though. I can of course copy files other ways like through Owncloud or something, but it's not as efficient if I want to copy several hundred files at once. Windows in VMware works though so I use that when I need that function.

Apps/Games I use

I kept hearing about how awesome the apps are and stuff.. My needs are pretty basic. I have a bunch of apps installed, but I have found that for the most part very few of them get used. Really I think the only application that is not included on the phone that I fire up more than once a day is Firefox. I use the built-in email client for work email, as well as the built in SMS client for text messages.

Other 3rd party apps I use on a semi regular basis

  • Nova launcher - I use this alternative launcher all the time, works very well.
  • Oceans HD live wallpaper - looks really nice
  • F-stop image gallery (seems to be pretty good, I like the dynamic albums it provides, I split my pictures up into portrait and landscape albums so I can get maximum viewing pixels without having to constantly flip the phone back and forth as I view the images)
  • MX Player (video player) works quite well too
  • Skype - roughly 80% of all work communications go through skype

Yet more 3rd apps I use on a less regular basis

  • K-9 Mail (used for personal email, when not traveling I fire it up maybe a couple times a week) - I use the built in email client for my work email(Exchange). Most of the time I just read personal email from a regular laptop or desktop in a webmail client.
  • Owncloud (access my colo server file storage)

Speaking of Owncloud, I am using DAVdroid (and the workaround) to sync contacts between the phone and my owncloud server, that is handy. I don't like the idea of sharing contacts with google or other service providers. The last time I stored contacts on exchange I forgot to take them off before I nuked my exchange account(when leaving the company) and I lost all of them so I decided that was not a good idea to try again. WebOS had a Synergy feature where it could integrate with the likes of LinkedIn directly to your contacts (and it had no ad tracking or anything it was pretty basic but it worked). I will not install the LinkedIn app for Android, too invasive.

As for games, I installed a few first person shooters and a Mech RTS game, I played the FPS games for about 2 minutes and haven't touched them since(sort of afraid my thumb is going to go through the screen with them). The Mech RTS game (MechCom) was pretty fun, though haven't touched it in about 3 weeks.

I have been playing the Simpsons Tapped out and Megapolis quite a bit, they are entertaining. Though I'd like to see a real Sim City game for Android(if there is one I haven't seen it). I poked around for a bunch of other apps/games but didn't see much that interested me. One thing I do note however is it seems like the Google play store could use a lot more categories, with so many apps/games it seems difficult to find something just by browsing around.

I have made sure to limit the apps based on the permissions, there are tons of apps out there that just want too many permissions and I won't take 'em. There's been quite a bit of talk about improving the permissions system of Android I do hope more work is done in that area especially being able to provide "fake" information to apps that are asking for too much. The phone came with the app (I think it came with it I might of downloaded it though) called Lookout Labs Ad Network Detector. Not sure how good it is but it scans all the apps and shows what the major categories of ad networks and what they do and what installed apps are using them. For me there are only 3 Ad Networks detected (Admob, Tapjoy and Millennial) and they don't collect a whole lot of info. Certainly I reject anything that wants to touch contacts, or take pictures, or send/read SMS, collect personal information etc..

I have a bunch more apps and some more games installed but they've all gotten minimal usage at this point.

Work related apps

One thing I could never do on the Pre3 was really anything work related outside of e-mail. Not a problem anymore.

  • Dell SonicWall VPN - while my main VPN is Citrix Access Gateway, there is no mobile app for that, I have Sonicwalls as well though(mainly used for site to site VPN). There is an Android (and IOS) app for them and it works quite well on Android.
  • Citrix XenApp Reciever - we have a very small XenApp server for operations purposes (some windows management software packages etc). This package(especially with the S-Pen for precision) works quite well on Android. I can fire up vCenter, or the 3PAR GUI tools(I don't use them much), or Firefox most recently I fired up Firefox to reconfigure our production load balancers(Citrix Netscaler) from my phone a few weeks ago. Being that the load balancers use Java applets those would not run directly on the phone(I don't think anyway).
  • iVMControl - vSphere interface though not very useful to me. Waaaay too slow to use over a 3-5,000 mile WAN connection. Much faster/easier/better to use XenApp and the regular vCenter client.
  • Microsoft Remote Desktop - haven't used this app yet, may not use it unless I have problems with XenApp, but it's there.
  • HP Storefront mobile access - interesting little app that grants me read only access into my 3PAR arrays. I don't need to login to them very often, but it's there if I need to view an alert or something.
  • HP Support - access to HP support cases. Only used it once to see what it did.
  • iLO Console - access to iLO I guess, doesn't seem too useful, I suppose if I want to access the console(can't remember the last time I had to do that), it doesn't seem to have an Android experience to access iLO functions for that it relies on the iLO web interface which I can otherwise just load in Firefox once I am on VPN.

I suppose the biggest thing I have NOT setup yet is SSH. I have a couple SSH clients installed but have not gone through setting them up with my keys(or generating new keys). None of my systems accept password authentication for SSH.  I was never able to SSH from my Palm phones so this is nothing new to me.

I have also not setup OpenVPN so I can VPN to my colo server. I have an OpenVPN client but it wants a config file in a special format that I haven't spent the time to figure out how to do yet. I did for a brief time have a command line OpenVPN client on my HP Touchpad but long since lost it. There were no Citrix, or Sonicwall or GUI OpenVPN clients that I was aware of for WebOS anyway.

GPS Navigation on Android

The first time I used mobile GPS navigation was back in I think it was 2001 with my Handspring Visor and a GPS Springboard expansion module along with a PalmOS GPS navigation app. It was fun, things have evolved a crazy amount since then.

Over the holidays I went on another road trip - covering just over 2,500 miles driving to Orange County, then to Tuscon, then to the Phoenix area and back home to the bay area. I was in my own car so I used the Kenwood/Garmin Stereo/Navigation system that I had installed just after I bought the car rather than the phone.

Picture of my car's trip meter from my 2013 holiday road trip.

Picture of my car's trip meter from my 2013 holiday road trip.

(thought this post could use some color so added the pic)

I did use the phone on a few occasions to find things, but did not use it for navigation itself. One thing I pretty quickly saw was lacking on the Android apps that at least I was using (which were Mapquest and Google maps) were two key functions that I frequently use on my car navigation:

  • Find places along my route (bonus points if you can limit the distance from the route, my car's nav system has some sort of default limit that is not adjustable)
  • Find places near my destination

Neither Google maps nor Mapquest seemed to have a similar function, which is too bad.  I'm sure you can do something similar with either perhaps just by zooming out along the route and searching, but that seems like more trouble than it should be.

I installed a bunch of other travel/road/traffic condition apps but I never used any of them on my trip (or since for that matter -- road conditions were fine anyway). My car nav system does not have any traffic info.

I'm going on another trip in March to Atlanta(to visit my company's colo for the first time in over two years), and probably will go to either Seattle or Washington DC as part of that trip, so I will certainly need navigation there as I don't know the area. At this point I've decided to take along a TomTom I bought a while back to do Navigation on that trip rather than rely on the phone. I used it on my last trip to DC and it worked well, I have a stand for it and it sits well on the dashboard etc. It also has the two functions above that I use quite frequently (though last time I was in DC the TomTom spent 30 minutes trying to convince me to go on a highway that was shut down for construction, that was frustrating ...)

I know there is a TomTom app for Android but after reading up on it I think for now I'll stick to the stand alone unit.

Conclusion

Overall I am very satisfied with the user experience and capabilities of my new Android phone. There is not much I miss from WebOS. I find the size & weight of the Note 3 to be very reasonable(more so than I was expecting). It performs well, and really gives me an order of magnitude more flexibility from a mobile perspective than I ever had on WebOS. I still do sort of wish I could of gotten a 64GB Note3, but it's not a huge deal, next time I guess!

I just ordered a Braven 710 bluetooth speaker (mainly for my upcoming trip), and that will likely be my first experience using NFC.

I guess that is enough writing for now.

30Jan/14Off

Facebook going in strong with Vertica

TechOps Guy: Nate

Came across this job post yesterday, thought it was interesting, somewhat on the heels of Facebook becoming a Vertica customer.

I still find it interesting at least given Facebook's history of in house solutions.

From the job posting

Do you like working with massive MPP databases? Are you interested in building one of the largest MPP data warehouses in the world? If yes, we want to talk to you. Facebook is seeking a Database Engineer to join the IT Engineering Infrastructure team to build the largest Vertica data warehouse in the world.

13Oct/13Off

Take a number: how to fix healthcare.gov

TechOps Guy: Nate

Sorry for slackin off recently, there just hasn't been a whole lot out there that has gotten me fired up.

Not too long ago I ranted a bit about outages. Basically saying if your site is down for a few hours, big whoop. It happens to everyone. The world is not going to end, your not going to go out of business.

Now if your website is down for a week or multiple weeks the situation is a bit different. I saw on a news broadcast that experts had warned the White House that the new $600M+ healthcare.gov web site was not ready. But the people leading the project, as it seems so typical probably figured the claims were overblown (are they ever? in my experience they have not been - though I've never been involved in a $600M project before, or anywhere close to it) and decided to press onwards regardless.

So they had some architecture issues, some load issues, capacity problems etc. I just thought to myself - this problem really sounds easy to solve from a technical standpoint. They tried to do this to some extent(and failed) apparently with various waiting screens. There are some recent reports that longer term fixes may take weeks to months.

I've been on the receiving end of some pretty poorly written/designed applications that it didn't really matter how much hardware you had it flat out wouldn't scale. I remember one situation in particular during an outage of some kind and the VP of Engineering interrupted us on the conference call and said Guys - is there anything I can buy that would make this problem go away?  The answer back to him was No. At this same company we had Oracle - obviously a big company in the database space come to our company and tell us they had no other customers in the world doing what we were doing, and they could not guarantee results. Storage companies were telling us the same thing. Our OLTP database at the time was roughly 8 times the next largest Oracle OLTP database in the world (which was Amazon). That was, by far the most over designed application I've ever supported. It was an interesting experience, I learned a lot. Most other applications that I have supported suffered pretty serious design issues, though none were quite as bad as this one company in particular.

My solution is simple - go old school, take a number and notify people when they can use the website.

Write a little basic app, point healthcare.gov to it, allow people to register with really basic info like name and email address (or phone# if they prefer to use SMS). This would be an entirely separate application not part of the regular web site. This is really light weight application, perhaps even store it in some noSQL solution(for speed) because worst case if you lose the data they'll just have to come back and register again.

So part of the registration the site would say we'll send you an email or SMS when your turn is up, with a code,  and you'll have a 24 hour window in which to use the site (past that and you have to register for a new number). If they can get the infrastructure done perhaps they could even have an automated phone system give them a call as well.

Then simply only allow a fraction of the # of people at a time on the website that the system can handle, if they built it for 50,000 people at a time I would probably start with 20,000 the first day or two and see how it goes(20,000 people per day not 20,000 simultaneous). Then ramp it up, if the application is scaling ok. As users register successfully the other application sees this and the next wave of notifications is sent. Recently I heard that officials were recommending people sign up through the call center(s), which I suppose is an OK stop gap but can't imagine the throughput is very high there either.

I figure it may take a team of developers a few days to come up with such an app.

Shift the load of people trying to hit an expensive application over and over again to a really basic high performance registration application, and put the expensive application behind a barrier requiring an authentication code.

IMO they should of done this from the beginning, perhaps even in advance generating times based on social security numbers or something.

All of this is really designed to manage the flood of initial registrations, once the tidal wave is handled then open the web site up w/o authentication anymore.

There should be a separate, static, high speed site(on many CDNs) that has all of the information people would need to know when signing up, again something that is not directly connected to the transactional system. People can review this info in advance and that would make sign ups faster.

Tagged as: Comments Off
18Sep/13Off

RIP Blackberry – Android is the Windows of the mobile world

TechOps Guy: Nate

You can certainly count me as in the camp of folks that believed RIM/Blackberry had a chance to come back. However more recently I no longer feel this is possible.

While the news today of Blackberry possibly cutting upwards of 40% of their staff before the end of the year, is not the reason I don't think it is possible, it just gave me an excuse to write about something..

The problem stems mainly from the incredibly fast paced maturation (can't believe I just used that word) of the smart phone industry especially in the past three years. There was an opportunity for the likes of Blackberry, WebOS, and even Windows Phone to participate but they were not in the right place at the right time.

I can speak most accurately about WebOS so I'll cover a bit on that. WebOS had tons of cool concepts and ideas, but they lacked the resources to put together a fully solid product - it was always a work in progress (fix coming next version). I felt even before HP bought them (and the feeling has never gone away even in the days of HP's big product announcements etc) - that every day that went by WebOS fell further and further behind(obviously some of WebOS' key technologies took years for the competition to copy, go outside that narrow niche of cool stuff and it's pretty deserted). As much as I wanted to believe they had a chance in hell of catching up again (throw enough money at anything and you can do it) - there just wasn't (and isn't) anyone willing to commit to that level - and it makes sense too - I mean really the last major player left willing to commit to that level is Microsoft - their business is software and operating systems.

Though even before WebOS was released Palm was obviously a mess when they went through their various spin offs, splitting the company divisions up, licensing things around etc. They floundered without a workable (new) operating system for many years. Myself I did not become a customer of Palm until I puchased a Pre back in 2009. So don't look at me as some Palm die hard because I was not. I did own a few Handspring Visors a long time ago and the PalmOS compatibility layer that was available as an App on the Pre is what drove me to the Pre to begin with.

So onto a bit of RIM. I briefly used a Blackberry back in 2006-2008 - I forget the model it was a strange sort of color device, I want to say monochrome-like color(I think this was it). It was great for email. I used it for a bit of basic web browsing but that was it - didn't use it as a phone ever. I don't have personal experience supporting BIS/BES or whatever it's called but have read/heard almost universal hatred for those systems over the years. RIM obviously sat on their hands too long and the market got away from them. They tried to come up with something great with QNX and BB10 but the market has spoken - it's not great enough to stem the tide of switchers, or to bring (enough) customers back to make a difference.

Windows Phone..or is it Windows Mobile.. Pocket PC anyone? Microsoft has been in the mobile game for a really long time obviously (it annoys me that press reporters often don't realize exactly how long Microsoft has been doing mobile -- and tablets for - not that they were good products but they have been in the market). They kept re-inventing themselves and breaking backwards compatibility every time. Even after all that effort - what do they have to show for themselves? ~3.5% global market share? Isn't that about what Apple Mac has ? (maybe Mac is a bit higher).

The mobile problem is compounded further though. At least with PCs there are (and have been for a long time) standards. Things were open & compatible. You can take a computer from HP or from Dell or from some local whitebox company and they'll all be able to run pretty much the same stuff, and even have a lot of similar components.

Mobile is different though, with ARM SoCs while having a common ancestor in the ARM instruction sets really seem to be quite a bit different enough that it makes compatibility a real issue between platforms. Add on top of that the disaster of the lack of a stable Linux driver ABI which complicates things for developers even more (this is in large part why I believe I read FirefoxOS and/or Ubuntu phone run on top of Android's kernel/drivers).

All of that just means the barrier to entry is really high even at the most basic level of a handset. This obviously wasn't the case with the standardized form factor components(and software) of the PC era.

So with regards to the maturation of the market the signs are clear now - with Apple and Samsung having absolutely dominated the revenues and profits in the mobile handset space for years now - both players have shown for probably the past year to 18 months that growth is really levelling out.

With no other players showing even the slightest hint of competition against these behemoths with that levelling of growth that tells me, sadly enough that the opportunity for the most part is gone now. The market is becoming a commodity certainly faster than I thought would happen and I think many others feel the same way.

I don't believe Blackberry - or Nokia for that matter would of been very successful as Android OEMs.  Certainly at least not at the scale that they were at - perhaps with drastically reduced workforces they could of gotten by with a very small market share - but they would of been a shadow of their former selves regardless. Both companies made big bets going it alone and I admire them for trying - though neither worked out in the end.

Samsung may even go out as well the likes of Xiaomi (never heard of them till last week) or perhaps Huawei or Lenovo coming in and butchering margins below where anyone can make money on the hardware front.

What really prompted this line of thinking though was re-watching the movie Pirates of Silicon Valley a couple of weeks ago following the release of that movie about Steve Jobs. I watched Pirates a long time ago but hadn't seen it since, this quote from the end of the movie really sticks with me when it comes to the whole mobile space:

Jobs, fresh from the launch of the Macintosh, is pitching a fit after realizing that Microsoft’s new Windows software utilizes his stolen interface and ideas. As Gates retreats from Jobs’ tantrum, Jobs screeches, “We have better stuff!

Gates, turning, simply responds, “You don’t get it. That doesn’t matter.

(the whole concepts really gives me the chills to think about, really)

Android is the Windows of the mobile generation (just look at the rash of security-related news events reported about Android..). Ironically enough the more successful Android is the more licensing revenue Microsoft gets from it.

I suppose in part I should feel happy being that it is based on top of Linux - but for some reason I am not.

I suppose I should feel happy that Microsoft is stuck at 3-4% market share despite all of the efforts of the world's largest software company. But for some reason I am not.

I don't know if it's because of Google and their data gathering stuff, or if it's because I didn't want to see any one platform dominate as much as Android (and previously IOS) was.

I suppose there is a shimmer of hope in the incorporation of the Cyanogen folks to become a more formalized alternative to the Android that comes out of Google.

All that said I do plan to buy a Samsung Galaxy Note 3 soon as mentioned before. I've severed the attachment I had to WebOS and am ready to move on.

23Aug/13Off

The world is ending: overreactions to outages

TechOps Guy: Nate

Paraphrasing from CNBC yesterday:

OMG!! CARL ICHAN IS TWEETERING ABOUT APPLE -- NASDAQ IS DOWN -- PEOPLE CAN'T TRADE ON THIS NEWS!!

Let me preface this a bit further and say in the line of work that I am in I have been on the receiving end of so many outages of various types ... some of them really nasty lasting hours, even down for multiple days, some involving some big data losses, many had me up for 20-30+ hours straight. Some of the most fun times I've had have been during big outages. Finally, some excitement!

My favorite outage that I can recall was one at AT&T about nine years ago. They were doing a massive migration to a new platform to support number portability, among other things. So they asked us to hold transactions in our queues while they were down for ~6-8 hours (the company I was at handled most of the mobile e-commerce for them at the time). So we did. 8 hours passed.. 10... 12... 16.. still down. No ETA .. It wasn't a huge deal for us for the first day, it became somewhat troublesome by the 3rd day as these queues were in memory and we had hard limits on memory(32-bit). But the folks in the AT&T stores were really hurting as they could not provision any new phones, all new orders had to be done on paper, then input into the computer system later. I forgot how long the outage was total I think around 4 days though. I looked at the whole situation and couldn't help but laugh. Lots of laughter. 8 hours to 4 days.. thousands of orders being placed via paper, by one of, if not the largest telcos in the world.

So I think I have a better perspective on this sort of thing than those less technical folk who freak out about stuff like the NASDAQ outage yesterday.

Taking NASDAQ specifically it was pretty absurd to see the whole situation unfold yesterday (I worked from home so saw the full thing end to end on CNBC). People coming on air and saying how they were frustrated that NASDAQ wasn't giving any information as to what was going on, speculation about complications being a public company and an exchange at the same time (and disclosure requirements etc). Then a bigwig comes on, Harvey Pitt, a former SEC chairman and just seems to ream NASDAQ, saying how it's totally unacceptable that they are down, there should be heavy fines and zero tolerance.

Come on folks - get a grip. It's a stock exchange. It's not a 911 system. People aren't going to die. If your system is so fragile that it can't survive a few hours of downtime on an exchange and can't tolerate a little volatility then it's your system that needs to be fixed.

You don't have control over the exchange, or the internet peering points between you and them(or your broker and them), there's so many points of failure that you should have a more robust system, the exchanges I have no doubt are incredibly complex, convoluted and obscure things that are constantly under attack by people trying to get trades through as quickly as possible, like those folks that manipulate the market.

Even the experts seem to be moving too fast, just barely a year ago Knight Capital lost $400 million in a matter of minutes due to a software bug. They later were forced to sell the company. More recently the almighty Goldman Sachs did something similar, last I saw they were hoping they would only lose $100 million as a result of that error.

Slow down, take a break. Things are moving too fast. I see people on CNBC constantly argue that the markets are really important because so many people have 401ks, IRAs etc. But reality doesn't agree with them. I don't recall the specific stat but I've heard it tossed around a few times something along the lines of 85% of stocks are owned by 5% of the population.

Another stat I've heard tossed around is  ~80% of the transactions these exchanges get today are from high frequency traders. So if HFT somehow goes away then these exchanges are in trouble revenue wise.

Those two stats alone tell me a lot about the state of the markets. I'm no financial expert obviously but I have watched CNBC a lot for many years now (going back to at least 2007) on a daily basis (RIP Mark Haines). I am often fascinated by the commentary, and the general absurdities of the market structure in general(I find in general it's more comedy than anything else). There's been very little investing going on for a very long time. Really the stock markets in general are outright gambling. Stocks rarely move on fundamentals anymore(not sure when they last did) it's all buzz, and emotions.

It's no wonder so many startups aren't interested in going IPO, and some other big established brands are wanting desperately to go private. To get away from activist investors, and the overall pressures to run your company in a pro-market fashion rather than what's best for the long term health of the company itself (and thus long term shareholders).

If your that dependent on market liquidity (e.g. the schemes folks like Lehman Brothers were doing rolling over their financials every night) - your doing something very wrong, and you deserve to get burned by it.

These exchanges are closed for upwards of 16 hours a day (and closed weekends and holidays!), there is some limited after hours trading, and some stocks trade on other exchanges as well, but the relative liquidity there is small.

This goes beyond NASDAQ of course and to outages in general. Whether it was the recent Google, or Amazon, or Microsoft, or whatever outages recently or others(I suffered through two yesterday myself that were the result of a 3rd party, one of which was literally minutes apart from when NASDAQ went down..).

So chill out. Fix the problem, don't rush or you might make a mistake and make things worse. Get it right, try not to let that particular scenario happen in the future.

It's just a website, it's just a stock exchange. It's not a nuclear reactor that is on the verge of melt down.

Breathe. The world isn't going to end because your site/business happens to be offline for a few hours.

Tagged as: Comments Off
21Aug/13Off

More IPv6 funnies…

TechOps Guy: Nate

Random, off topic, boring post but I felt compelled to write it after reading a fairly absurd comment on slashdot from another hard core IPv6 fan.

Internet hippies at it again!

I put the original comments in italics, and the non italic stuff is the IPv6 person responding. I mean honestly I can't help but laugh.

I was a part of the internet when it started and was the wild wild west.  Everyone had nearly unlimited ip addresses and NOBODY used them for several reasons. First nobody put everything on the internet.

That was then. Now is now. The billion people on Facebook, Twitter, Flickr don't put anything online? Sure, it's all crap, but it sure is not nothing.

It's just Dumb to put workstations on the internet... Sally in accounting does not need a public IP and all it does is make her computer easier to target and attack. Hiding behind that router on a separate private network is far more secure. Plus it is easier to defend a single point of entry than it is to defend a 255.255.0.0 address space from the world.

Bullsh*t. If in IPv4 your internal network would be 192.168.10.0/24, you can define an IPv6 range for that as well, e.g. 2001:db8:1234:10::/72. And then you put in your firewall:

2001:db8:1234:10::/72 Inbound: DENY ALL

Done. Hard? No. Harder than IPv4? No. Easier? Yes. Sally needs direct connection to Tom in the other branch (for file transfer, video conference, etc):

2001:db8:1234:10::5411/128 Inbound: ALLOW ALL FROM 2001:db8:1234:11::703/128

Good luck telling your IPv4 CGN ISP you need a port forwarded.

Second I have yet to have someone give me a real need for having everything on the internet with a direct address. you have zero need to have your toaster accessible from the internet.

Oh yeah? Sally might need that 30 GB Powerpoint presentation of her coworker in the other branch. Or that 100 MB customer database. Well, you know, this [xkcd.com]. How much easier would that be with a very simple app that even you could hack together that sends a file from one IP address to the other. Simple and fast, with IPv6. Try it with IPv4.

It's amazing to me how folks like this think that everything should just be directly connected to the internet. Apparently this IPv6 person hasn't heard of a file server before, or a site to site VPN. Even with direct accessibility I would want to enforce VPN between the sites, if nothing else to not have to worry that any communications would not be encrypted (or in some cases WAN optimized). Same goes for remote workers - if your at a remote location and wanting to talk to a computer on the corporate LAN or data center - get on VPN. I don't care if you have a direct route to it or not (in fact I would ensure you did not so you have no choice).

The problems this person cites have been solved for over a decade.

I'm sorry but anyone that argues that 2001:db8:1234:10::5411/128 is simpler than 192.168.10.0/24 is simpler is just ...not all there.

The solutions perhaps may not be as clean as something more native, though the thought of someone wanting to move 30GB of data over anyone's internet connection at the office would be a very bad thing to do without arranging something with IT first (do it off hours, throttle it, something).

The point is the solutions exist, and they work. Fact remains that if you go native IPv6 your going to have MUCH MORE PAIN than any of the hacks that you may have to do with IPv4 today. IPv6 fans fail to acknowledge that up front. They attack IPv4/NAT/etc and just want the world to turn the switch off of IPv4 and flip everyone over.

I have said for years I don't look forward to IPv6 myself (mainly for the numbering scheme, it sucks hard). If the time comes where I need IPv6 for myself or the organization I work for there are other means to get it (e.g. NAT - at the load balancer level in my case) that will work for years to come (until perhaps there is some sort of mission critical mass of outbound IPv6 connectivity that I need - I don't see that in the next 5-8 years - beyond that who knows maybe I won't be doing networking anymore so won't care).

I'm sure people like me are the kind of folks IPv6 people hate. I don't blame 'em I suppose.

There is nothing - absolutely nothing that bugs me about IPv4 today. Not a damn thing hinders me or the organizations I have worked for. At one point SSL virtual hosting was an issue, but even that is solved with SNI (which I just started using fairly recently actually).

The only possibility of having an issue I think is perhaps if my organization merged with another and there was some overlapping IP space. Haven't personally encountered that problem though in a very long time (9 years - and even then we just setup a bunch of 1:1 NATs I think - I wasn't the network engineer at the time so wasn't my problem).

I remember one company I worked for 13 years ago - they registered their own /24 network back in the early 90s, because the people at the time believed they had to in order to run an internal network. The IP space never got used (to my knowledge) and it was just lingering around - the contact info was out of date and we didn't have any access to it (not that we needed it, was more a funny story to tell).

When I set this server up at Hurricane Electric, one of the things they asked me was if I wanted IPv6 connectivity, since they do it natively I believe (one of the biggest IPv6 providers out there I think globally ?). I thought about it for a few seconds and declined, don't need it.

IPv6 fans need to come up with better justification for the world to switch other than "the internet is peer to peer and everyone needs a unique address" (because that reason doesn't cut it for folks like me, and given the world's glacial pace of migration I think my view is the norm rather than the exception). I've never really cared about peer to peer anything. The internet in general has been client-server and will likely remain so for some time (especially given the average gap between download and upload bandwidth on your typical broadband connection)

Given I have a server with ~3.6TB of usable space on a 100Mbps unlimited bandwidth connection less than 25 milliseconds from my home I'd trade download bandwidth for upload bandwidth in a HEARTBEAT - I'd love to be able to get something like 25/25Mbps unfortunately the best upload i can get is 5Mbps - while I can get 150Mbps down -- my current plan is more like 2Mbps up and 16Mbps down.

Speedtest.net results for this server. I had to try several different test servers before I found one that was fast enough to handle me.

Speedtest.net results for this server. I had to try several different test servers before I found one that was fast enough to handle me.

ANYWAY........ I had a good laugh at least.

Back to your regularly scheduled programming..

Tagged as: Comments Off
17Aug/13Off

Yahoo! yanks! website! of! person! who! documented! his! suicide!

TechOps Guy: Nate

The only thing technical related to this is the fact that Yahoo! yanked the guy's site. I suppose I can understand why, but I am very glad that the site(at least for the moment) lives on at a mirror.

I just came across this on slashdot. The discussion wasn't all that interesting but I have been reading a mirror of the website, and have to say it is quite an amazing write up.

I felt this guy took the time to think and write about his thoughts and did the world a favor in showing his state of mind. So the least I could do is read it - and perhaps comment on it a bit (with whatever respect I can give).

From what I can tell he committed suicide because he felt his mind was going, he was no longer (as) productive to society as he wanted to be, and he had a very negative outlook on near term civilization as we know it.

He took his life two days ago, on his 60th birthday in a police parking lot with a self inflicted gunshot wound to the head.

I have no idea who Martin Manley was but it was very interesting to see his line of thinking.

Some good quotes

I began seeing the problems that come with aging some time ago. I was sick of leaving the garage door open overnight. I was sick of forgetting to zip up when I put on my pants. I was sick of forgetting the names of my best friends. I was sick of going downstairs and having no idea why. I was sick of watching a movie, going to my account on IMDB to type up a review and realizing I've already seen it and, worse, already written a review! I was sick of having to dig through the trash to find an envelope that was sent to me so I could remember my own address - especially since I lived in the same place for the last nine years!

[..]

I didn’t want to die alone. I didn’t want to die of old age. I didn’t want to die after years of unproductivity. I didn’t want to die having my chin and my butt wiped by someone who might forget which cloth they used for which. I didn’t want to die of a stroke or cancer or heart attack or Alzheimer’s. I decided I was gettin’ out while the gettin’ was good and while I could still produce this website!

He does mention a life insurance policy that expires next year, and he wouldn't of been able to afford to renew it.  So that money can go to the folks he cares about.  Though I thought most, if not all such policies excluded suicide. I am not sure though, never looked into it. He seems intelligent enough that he would of known the details of the policy he had.

I felt pretty good about being prepared for economic collapse – the primary reason being all the gold and silver I owned. But, then one day I realized that all the gold and silver and guns and ammo and dried food and toilet paper in the world wouldn’t prevent me from seeing the calamity with my own eyes - either ignoring other's plight or succumbing to it. And, that’s something I decided I simply was not willing to live through.

Right with him there, except for the fact about being prepared. I acknowledged a long time ago that there's no point in trying to prepare for such an event, the resources required would be pretty enormous. My best friend(who is reading this, HI!!!)  has told me on a couple of occasions to go live with him in a cabin in the woods, live off the land(in the event of total collapse).. Not feasible for various reasons I don't want to get into here.

But, if you plan to stick around, then you better plan to watch an economic collapse that will be worse than anything you can imagine.

It's frustrating to me to see all of our leaders, whether it is in corporations or government show such, I'm not sure what the words are other than to call it something like false confidence.  Hiding the truth because sentiment is such an important factor of the economy. It's everywhere, the more I see folks talk the more I see in most cases they really have no idea what they are doing, they are just hoping it works out.

The more I learn the more I realize how young of a civilization we really are and how little we actually know.

What pisses me off more than anything is the system in place that tries to educate us so we think we know. So we have faith in those that are making the decisions.

I'm the first one to admit I don't know what the answers are(macro global economic/political type things) -- but I'm also (one of)the first ones to admit that uncertainty in the first place, which would probably make me a bad leader. I can't portray confidence because I don't have confidence(in that stuff anyway, I believe I do portray confidence when it comes to the tech things I work with), I do have honesty, which is what this Martin fellow seems to have a lot of as well. Most people don't want the truth, they just want to feel good.

One of the only videos I ever uploaded to Youtube was this, which is a good illustration from the corporate side of things. There is another bit (haven't been able to find it last I checked) which showed the same sort of thing from our previous president where they walked through his various descriptions of the impending economic downfall. From storm clouds to whatever it was in the end.

I'm in complete agreement with this Martin guy though, what we experienced in 2008-2010 or whatever is nothing compared to what is coming. When that is exactly I'm not sure, it seems folks like the Fed etc seem to be able to pull rabbits out of their hats to drove the ponzi scheme just a bit further. My general expectation is within the next two decades, and I think that is probably a conservative estimate.

It is unfortunate that the topic of suicide is amongst those topics that are considered taboo. People don't talk about it. The common theme is often mention the word and your deemed crazy and they want to lock you up in a padded room, fill you with meds until you conform, or die in the process.

There was a news report on NBC that I saw last year about a facility(hospital) where they assist people and their families to prepare for when that day will come. People perhaps like Martin who don't want to live out their lives as a burden to others, unable to mentally and/or physically perform things. I thought it was pretty amazing to see. They go through tons of questions with the patient about scenarios and what do do with those scenarios. So when the time comes there is no doubt. There won't be some random family member saying KEEP THEM ALIVE I DON'T CARE IF IT COSTS A MILLION $.

As with many topics, there was (in retrospect, it really didn't hit me until a few years ago) an excellent Star Trek: The Next Generation episode on this very topic. It was called Half a Life from Season 4 (1991). At the time when I saw it, I suppose you could say I didn't understand it, as I found the episode quite boring (well the special effects in the beginning were pretty neat). But as the health care debate picked up a few years ago I realized that episode told an interesting story that I never bothered to realize until that time.

Martin lists reasons he considered not to commit suicide (paraphrasing(?) them briefly, see the site for more details):

  1. Loved ones - obvious, people will miss you. More importantly perhaps is if you are in a situation where you are supporting someone else and they are dependent upon you. Martin was not in this situation
  2. Want to see the future. Live out retirement, travel the world perhaps, read books, play backgammon, look at granny porn (my grandfather did this a lot in the years before he died, honestly I did not know such porn existed until my sister+mother told me)
  3. People want to accomplish as much as they can in their lives and they don’t want to run out of time before they do it. Of course, for people who think that way, they never fulfill all those accomplishments anyway and they never will. So, the only thing to do is keep chasing them until you die.
  4. The last major reason I thought of for why people want to live indefinitely is the whole notion of leaving a legacy.

More quotes..

I once had a quasi bucket list when I was about 22 – things to accomplish by the time I was 30. When 30 came around and I hadn’t accomplish them, I decided the bucket list idea was stupid.

[..]

There will always be reasons to want to stay alive another year or five years or 10 years. It wouldn’t have mattered how long I lived, there would have been hundreds or thousands of itches to scratch!

[..]

I could take pride in the fact that I wasn’t going to be sucking on the nipple of the federal debt by taking social security and medicare. When the US economy collapses, it won't have been me that contributed to taking it down.

Here he touches on life insurance again, I guess they do pay out for suicide -

Another reason why age 60 is ideal is that my life insurance expires next year and I would not be able to afford to get new insurance without paying a ton. And, it requires two years of waiting - once you get insurance - before you can commit suicide and still have the beneficiaries receive the death benefit.

Holy sh*t, he even brings up the aforementioned Half a Life episode of TNG. He devotes several paragraphs to it!

Then he goes into how he did it - a gun

One of the problems with shooting oneself is the obvious mess. I thought about that a lot. I didn’t want anyone I knew discovering my body and I didn’t want to make a mess in the house – something my sister or my landlord would have to deal with. No way.

[..]

I finally decided the best way to do it would be at 5AM on August 15, 2013 at the far southeast end of the parking lot at the Overland Park Police Station. If everything worked out right – and I’m sure it did, I called 911 at 5AM. I told them “I want to report a suicide at the south end of the parking lot of the Overland Park Police Station at 123rd and Metcalf. Bang.”

He left a note on himself which in part read

“I committed suicide of my own free will. I am not under the influence of any drugs. I am sorry for your inconvenience! You will be contacted within a matter of hours by my sister. She will find out about this by an overnight letter and/or email I sent to her which she will get this morning. In it, I explain the exact location where I shot myself and gave her your phone number. At that time, she will tell you who I am. If you discover who I am prior to her call, please do not contact her. I do not want her (or anyone else I sent letters to overnight) to find out about it from you. I want them to find out about it from me. Thank you!

Another quote

The act of suicide can be horrible for those left behind. I couldn’t control the fact of the matter, but I could control the circumstances. I believe the way I did it, coupled with the overnight letters/emails and this web site, is the best I can do to mitigate the hurt.

He wrote a bunch more but the most interesting stuff I suspect was in the first few pages that I read (up until the "Gun Control" topic which is only semi related).

It is fascinating to me to see the level of thought and analysis that went into his decision. Whether you agree with the decision or not is up to you, but to me I think the point is it was his decision. Call him greedy, or selfish or crazy or whatever, but I firmly believe that having the "freedom" (though I think it is illegal(hence the quotes), which is crazy) to make this decision and follow through on it is an important right to have.

Now if you cause major harm as a result of your action (say perhaps drive down the wrong direction on a freeway) then that's a different scenario.  Martin was incredibly thoughtful in how he handled the whole situation, and for that I ..well I don't really know how to put that into words.

If you are someone who would want to rob someone else of this right then I'd say it is you who are greedy, selfish or crazy. The only exception would be again if you were somehow economically dependent upon them.

While I understand Yahoo! decision to yank the site, as it probably was against their TOS - I only wish Martin would of chosen a better place to host the site! Fortunately there is a mirror, I'll probably snag a copy of it myself just in case.

I don't consider myself a very emotional person, reading his writing really did not evoke any emotional response. I went to my first(thus far, only) funeral almost two years ago now for a cousin of mine who also committed suicide(also via self inflicted gunshot). I hadn't seen him since the late 80s, I had no emotional response for that either. I did feel bad for not "feeling" bad though, as strange as that may sound to write. I sat through the funeral as his loved ones and friends told stories about him and stuff for a few hours. It was an interesting experience. I'm sorry he is gone but from what I knew of the broader situations the family had experienced over the past few decades I can totally understand his decision. Anyway that is sort of off topic.

I hope the folks that where closest to Martin understand, accept, and most importantly support his decision.

I suppose the obvious thing to say here is may he rest in peace.

Any further discussion on the topic I am more than happy to talk about off line.

Tagged as: Comments Off
10Aug/13Off

The Myth of online backup and the future of my mobility

TechOps Guy: Nate

I came across this article on LinkedIn which I found very interesting. The scenario given by the article was a professional photographer had 500GB of data to backup and they decided to try Carbonite to do it.

The problem was Carbonite apparently imposes significant throttling on the users uploading large amounts of data -

[..]At that rate, it takes nearly two months just to upload the first 200GB of data, and then another 300 days to finish uploading the remaining 300GB.

Which takes me back to a conversation I was having with my boss earlier in the week about why I decided to buy my own server and put it in a co-location facility, instead of using some sort of hosted thing.

I have been hosting my own websites, email etc since about 1996. At one point I was hosted on T1s at an office building, then I moved things to my business class DSL at home for a few years, then when that was no longer feasible I got a used server and put it up at a local colo in Seattle. Then I decided to retire that old server(build in 2004) and spent about a year in the Terremark vCloud, before buying a new server and putting it up at a colo in the Bay area where I live now.

My time in the Terremark cloud was OK, my needs were pretty minimal, but I didn't have a lot of flexibility(due to the costs). My bill was around $120/mo or something like that for a pair of VMs. Terremark operates in a Tier 4 facility and doesn't use the built to fail model I hate so much, so I had confidence things would get fixed if they ever broke, so I was willing to pay some premium for that.

Cloud or self hosting for my needs?

I thought hard about whether or not to invest in a server+colo again or stay on some sort of hosted service. The server I am on today was $2,900 when I bought it, which is a decent amount of money for me to toss around in one transaction.

Then I had the idea of storing data off site, I don't have much that is critical, mostly media files and stuff that would take a long time to re-build in case of major failure or something. But I wanted something that could do at least 2-3TB of storage.

So I started looking into what this would cost in the cloud. I was sort of shocked I guess you could say. The cost for regular, protected cloud storage was going to easily be more than $200/mo for 3TB of usable space.

Then there are backup providers like Carbonite, Mozy, Backblaze etc.. I read a comment on Slashdot I think it was about Backblaze and was pretty surprised to then read their fine print -

Your external hard drives need to be connected to your computer and scanned by Backblaze at least once every 30 days in order to keep them backed up.

So the data must be scanned at least once every 30 days or it gets nuked.

They also don't support backing up network drives. Most of the providers of course don't support Linux either.

The terms do make sense to me, I mean it costs $$ to run, and they advertise unlimited. So I don't expect them to be storing TBs of data for only $4/mo. It just would be nice if they (and others) would be more clear on their limitations up front, at least unlike the person in the article above I was able to make a more informed decision.

The only real choice: Host it myself

So the decision was really simple at that point. Go invest and do it myself. It's sort of ironic if you think about it, all this talk about cloud saving people money. Here I am, just one person, with no purchasing power whatsoever and I am saving more money doing it myself then some massive scale service provider can offer it.

The point wasn't just the storage though. I wanted something to host:

  • This blog
  • My email
  • DNS
  • my other websites / data
  • would be nice if there was a place to experiment/play as well

So I bought this server which is a single socket quad core Intel chip, originally with 8GB, now it has 16GB of memory, and 4x2TB SAS disks in RAID 1+0(~3.6TB usable) w/3Ware hardware RAID controller(I've been using 3Ware since 2001). It has dual power supplies(though both are connected to the same source, my colo config doesn't offer redundant power). It even has out of band management with full video KVM and virtual media options. Nothing like the quality of HP iLO, but far better than what a system of this price point could offer going back a few years ago.

On top of that I am currently running 5 active VMs

  • VM #1 runs my personal email, DNS,websites, this blog etc
  • VM #2 runs email for a few friends, and former paying customers(not sure how many are left) from an ISP that we used to run many years ago, DNS, websites etc
  • VM #3 is a OpenBSD firewall running in layer 3 mode, also provides site to site VPN to my home, as well as a end-user VPN for my laptop when I'm on the road)
  • VM #4 acts as a storage/backup server for my home data with a ~2TB file system
  • VM #5 is a windows VM in case I need one of those remotely. It doesn't get much use.
  • VM #6 is the former personal email/dns/website server that ran a 32-bit OS. Keeping it around on an internal IP for a while in case I come across more files that I forgot to transfer.

There is an internal and an external network on the server, the site to site VPN of course provides unrestricted access to the internal network from remote which is handy since I don't have to rely on external IPs to run additional things. The firewall also does NAT for devices that are not on external IPs.

Obviously as you might expect the server sits at low CPU usage 99% of the time and it's running at around 9GB of used memory, so I can toss on more VMs if needed. It's obviously a very flexible configuration.

When I got the server originally I decided to host it with the company I bought it from,  and they charged me $100/mo to do it. Unlimited bandwidth etc.. good deal(also free on site support)!  First thing I did was take the server home and copy 2TB of data onto it. Then I gave it back to them and they hosted it for a year for me.

Then they gave me the news they were going to terminate their hosting and I had only two weeks to get out. I evaluated my options and decided to stay at the same facility but started doing business with the facility itself (Hurricane Electric). The down side was the cost was doubling to $200/mo for the same service (100Mbit unlimited w/5 static IPs), since I was no longer sharing the service with anyone else. I did get a 3rd of a rack though, not that I can use much of it due to power constraints(I think I only get something like 200W). But in the grand scheme of things it is a good deal, I mean it's a bit more than double what I was paying in the Seattle area but I am getting literally 100 times the bandwidth. That gives me a lot of opportunities to do things. I've yet to do much with it beyond my original needs, that may change soon though.

Now granted it's not high availability, I don't have 3PAR storage like Terremark did when I was a customer, I have only 1 server so if it's down everything is down.  It's been reliable though, providing really good uptime over the past couple of years. I have had to replace at least two disks, and I also had to replace the USB stick that runs vSphere the previous one seemed to have run out of flash blocks as I could no longer write much to the file system. That was a sizable outage for me as I took the time to install vSphere 5.1 (from 4.x) on the new USB stick, re-configure things as well as upgrade the memory all in one day, took probably 4-5 hours I think. I'm connected to a really fast backbone and the network has been very reliable (not perfect, but easily good enough).

So my server was $2,900, and I pay currently $2,400/year for service. It's certainly not cheap, but I think it's a good deal still relative to other options. I maintain a very high level of control, I can store a lot of data, I can repair the system if it breaks down, and the solution is very flexible, I can do a lot of things with the virtualization as well as the underlying storage and the high bandwidth I have available to me.

Which brings me to next steps, something I've always wanted to do is make the data more mobile, that is one area which it was difficult(or impossible) to compete with cloud services, especially on things like phones and tablets. Since they have the software R&D to make those "apps" and other things.

I have been using WebOS for several years now, which of course runs on top of Linux. Though the underlying Linux OS is really too minimal to be of any use to me. It's especially useless on the phone where I am just saddened that there has never been a decent terminal emulation app released for WebOS. Of all the things that could be done, that one seems really trivial. But it never happened(that I could see, there were a few attempts but nothing usable as far as I could tell). On the touchpad things were a little different, you could get an Xterm and it was kind of usable, significantly more so than the phone. But still the large overhead of X11 just to get a terminal seemed quite wasteful. I never really used it very much.

So I have this server, and all this data sitting on a fast connection but I didn't have a good way to get to it remotely unless I was on my laptop (except for the obvious like the blog etc are web accessible).

Time to switch to new mobile platform

WebOS is obviously dead(RIP), in the early days post termination of the hardware unit at HP I was holding out some hope for the software end of things but that hope has more or less dropped to 0 now, nothing remains but disappointment of what could of been. I think LG acquiring the WebOS team was a mistake and even though they've announced a WebOS-powered TV to come out early next year, honestly I'll be shocked if it hits the market. It just doesn't make any sense to me to run WebOS on a TV outside of having a strong ecosystem of other WebOS devices that you can integrate with.

So as reality continued to set in, I decided to think about alternatives, what was going to be my next mobile platform. I don't trust Google, don't like Apple. There's Blackberry and Windows Phone as the other major brands in the market. I really haven't spent any time on any of those devices. So I suppose I won't know for sure but I did feel that Samsung had been releasing some pretty decent hardware + software (based on stuff I have read only), and they obviously have good market presence.  Some folks complain etc.. If I were to go to a Samsung Android platform I probably wouldn't have an issue. Those complaining about their platform probably don't understand the depression that WebOS has been in since about 6 months after it was released - so really anything relative to that is a step up.

I mean I can't even read my personal email on my WebOS device without using the browser. Using webmail via the browser on WebOS for me at least is a last resort thing, I don't do it often(because it's really painful - I bought some skins for the webmail app I use that are mobile optimized only to find they are not compatible with WebOS so when on WebOS I use a basic html web mail app, it gets the job done but..). The reason I can't use the native email client is I suppose in part my fault, the way I have my personal email configured is I have probably 200 email addresses and many of them go directly to different inboxes. I use Cyrus IMAP and my main account subscribes to these inboxes on demand. If I don't care about that email address I unsubscribe and it continues to get email in the background. WebOS doesn't support accessing folders via IMAP outside of the INBOX structure of a single account. So I'm basically SOL for accessing the bulk of my email (which doesn't go to my main INBOX). I have no idea if Samsung or Android works any different.

The browser on the touchpad is old and slow enough that I keep javascript disabled on it, I mean it's just a sad decrepit state for WebOS these days(and has been for almost two years now). My patience really started running out recently when loading a 2-page PDF on my HP Pre3, then having the PDF reader constantly freeze (unable to flip between pages, though the page it was on was still very usable) if I let it sit idle for more than a couple of minutes (have to restart the app).  This was nothing big, just a 2-page PDF the phone couldn't even handle that.

I suppose my personal favorite problem is not being able to use bluetooth and 2.4Ghz wifi at the same time on my phone. The radios conflict, resulting in really poor quality over bluetooth or wifi or both. So wifi stays disabled the bulk of the time on my phone since most hotspots seem only to do 2.4Ghz, and I use bluetooth almost exclusively when I make voice calls.

There are tons of other pain points for me on WebOS, and I know they will never get fixed, those are just a couple of examples. WebOS is nice in other ways of course, I love the Touchstone (inductive charging) technology for example, the cards multitasking interface is great too(though I don't do heavy multi tasking).

So I decided to move on. I was thinking Android, I don't trust Google but, ugh, it is Linux based and I am a Linux user(I do have some Windows too but my main systems desktops, laptops are all Linux) and I believe Windows Phone and BlackBerry would likely(no, certainly) not play as well with Linux as Android. (WebOS plays very well with Linux, just plug it in and it becomes a USB drive, no restrictions - rooting WebOS is as simple as typing a code into the device). There are a few other mobile Linux platforms out there, I think Meego(?) might be the biggest trying to make a come back, then there is FirefoxOS and Ubuntu phone.. all of which feel less viable(in today's market) than WebOS did back in 2009 to me.

So I started thinking more about leaving WebOS, and I think the platform I will go to will be the Samsung Galaxy Note 3, some point after it comes out(I have read ~9/4 for the announcement or something like that). It's much bigger than the Pre3, not too much heavier(Note 2 is ~30g heavier). Obviously no dedicated keyboard, I think the larger screen will do well for typing with my big hands. The Samsung multimedia / multi tasking stuff sounds interesting(ability to run two apps at once, at least Samsung apps).

I do trust Samsung more than Google, mainly because Samsung wants my $$ for their hardware. Google wants my information for whatever it is they do..

I'm more than willing to trade money in a vein attempt to maintain some sort of privacy. In fact I do it all the time, I suppose that could be why I don't get much spam to my home address(snail mail). I also very rarely get phone calls from marketers(low single digits per year I think), even though I have never signed up to any do not call lists(I don't trust those lists).

Then I came across this comment on Slashdot -

Well I can counter your anecdote with one of my own. I bought my Galaxy S3 because of the Samsung features. I love multi-window, local SyncML over USB or WiFi so my contacts and calendar don't go through the "cloud", Kies Air for accessing phone data through the browser, the Samsung image gallery application, the ability to easily upgrade/downgrade/crossgrade and even load "frankenfirmware" using Odin3, etc. I never sign in to any Google services from my phone - I've made a point of not entering a Google login or password once.

So, obviously, I was very excited to read that.

Next up, and this is where the story comes back around to online backup, cloud, my co-lo, etc.. I didn't expect the post to be this long but it sort of got away from me again..

I think it was on another Slashdot comment thread actually (I read slashdot every day but never have had an account and I think I've only commented maybe 3 times since the late 90s), where someone mentioned the software package Owncloud.

Just looking at the features, once again got me excited. They also have Android and IOS apps. So this would, in theory, from a mobile perspective allow me to access files, sync contacts, music, video, perhaps even calendar(not that I use one outside of work which is Exchange) and keep control over all of it myself. Also there are desktop sync clients (ala dropbox or something like that??) for Linux, Mac, and Windows.

So I installed it on my server, it was pretty easy to setup, I pointed it to my 2TB of data and off I went. I installed the desktop sync client on several systems(Ubuntu 10.04 was the most painful to install to, had to compile several packages from source but it's nothing I haven't done a million times before on Linux). The sync works well (had to remove the default sync which was to sync everything, at first it was trying to sync the full 2TB of data, and it kept failing, not that I wanted to sync that much...I configured new sync directives for specific folders).

So that's where I'm at now. Still on WebOS, waiting to see what comes of the new Note 3 phone, I believe I saw for the Note 2 there was even a custom back cover which allowed for inductive charging as well.

It's sad to think of the $$ I dumped on WebOS hardware in the period of panic following the termination of the hardware division, I try not to think about it ..... The touchpads do make excellent digitial picture frames especially when combined with a touchstone charger.  I still use one of my touchpads daily(I have 3), and my phone of course daily as well. Though my data usage is quite small on the phone since there really isn't a whole lot I can do on it, unless I'm traveling and using it as a mobile hot spot.

whew, that was a lot of writing.

24Jul/13Off

Opscode is learning

TechOps Guy: Nate

A few months ago I wrote a long rant on how Opscode has a lot to learn about running operations.

Problems included:

  • status.opscode.com site returning broken HTTP Location headers which broke standards compliant browsers (I had reported this issue to them through multiple channels and support tickets for a good 7 months)
  • Taking scheduled downtime in the middle of a business day

It APPEARS someone read that post, because recently the status site was fixed. It now redirects to opscode.tumblr.com and since that time I have seen no issues with the site.

Also I see they have a scheduled downtime for some of their databases and they are scheduling it for 9PM Pacific time (Opscode is HQ'd in Pacific time), instead of say one in the afternoon. Obviously people in far time zones may not like that as much, but it makes sense to their U.S. customers(which I'd imagine is the bulk of their customer base but I don't know).

They've also gone through some effort to post analysis on outages/performance issues recently as well which is nice.

I have two remaining requests, in case Opscode is reading:

  • Schedule downtime further in advance, the most recent announcement provides about 48 hours of notification, I think it'd be better to provide one week notice. Take a look at what other service providers do for planned outages, my experience says 48 hours is not sufficient notice for scheduled downtime. If it's an emergency, then obviously a shorter window is acceptable just say it's an emergency and try to explain why it's an emergency.
  • Provide actual dates and times for the posts on the status site. Now it just says things like "17 hours ago" or "5 days ago".
  • Be consistent on the time zone used. Some posts use UTC, others(scheduled events) refer to Pacific time. I don't care which myself (well honestly I prefer Pacific since I am in that zone, but I can understand using UTC too).
  • Provide pro-active notification of any customer impacting maintenance.  Maybe all of their other customers follow them on twitter, I don't know. I don't use twitter myself. So having an email notification option (perhaps opt in by default) to customer addresses registered with the platform for such things would be good to consider.

Now as for Chef, there's tons of things that could be improved with Chef to make it easier to use.. My latest issue is whenever I pull up the JSON to edit an environment, or a node or whatever the order is not consistent. My co-worker says the data is not ordered, and it has never been consistent for him, for me the issues just started a few weeks/month or two ago. It's quite annoying. For example if I want to change the role of a node, I would knife node edit <hostname>, then skip to the end of the file, and change the role.  Now sometimes the role is at the top of the file, other times it is at the bottom (it's never shown up as in the middle).

Pick a way to display the information and display it consistently! How hard is that to do.. It's not as if I can pipe the JSON to the sort command and have it sort for me. I've never liked JSON for that reason -- my saying is If it's not friendly with grep and sed it's not a friend to me. Or something like that ..  JSON seems to be almost exactly the opposite of what Linux admins want to deal with, it's almost as bad as binary data, I just hate it. If I don't have to deal with it (e.g. it's used in applications and I never see it) - fine go nuts. Same goes for XML. I used to support a big application whose developers were gung ho for XML config files, we literally had several hundred. It would take WEEKS (literally) of configuration auditing(line by line) prior to deployment - and even then there was still problems. It was a massive headache. Using JSON just brings me back to those days.

The syntax is so delicate as well, one extra comma, or missing quote or anything the whole thing blows up(it wouldn't be so bad if the tool ran a simple syntax check and pointed out what the error was and returned you to the editor to fix it telling you what line it was on, but in this case it just bombs and you lose all your changes -- Opscode folks - look at visudo - it does this better..)

The only thing worse(off the top of my head) than the syntax for the chef recipes itself, is the JSON.. or maybe that should be vise versa..

Opscode and Chef are improving I guess is the point, maybe in the next decade or so it will become a product that I would consider usable to mere mortals.

Tagged as: Comments Off
5Jun/13Off

Real life Sim City

TechOps Guy: Nate

[WARNING: Non technical content directly ahead]

I've looked at Google maps a lot over the years, but don't remember ever seeing something quite like this. I played tons of Sim City many years ago and when I saw this I was immediately reminded of Sim City. It just seems so ..familiar.

I'm planning on staying at a hotel in this town in Nevada in a couple of weeks to visit a friend who is coming in from out of town(in case you were wondering how I stumbled upon this).

This first picture reminds me of many times when I would build out a neighborhood in Sim City with the roads, zone it with light (or medium) residential, perhaps  put neighborhood school near by - then watch the houses pop up one by one:

Real life Sim City Part 1

You can see a few individual houses here and there, and it's pretty easy to make out what look a lot like Sim City zoned plots of land(semi square shaped), obviously with a bunch of roads that are already complete. For the most part very clean empty plots of land. Much different than what I have seen many times in the past where perhaps there is a big real estate project under development and many houses are being built simultaneously with the road being laid out.

There is another part of the town that is quite similar, again eerily reminds me of Sim City:

Sim City in real life part 2

In this case I'm again reminded of some low density residential, along with a park in the middle(well in this case the other half of the middle is not yet laid down (not in the picture above, see the google maps link). The plots are so uniform, the houses remind me so much of Sim City.

Tagged as: Comments Off