I don’t visit the MySQL performance blog too often, but today happened to run across a very interesting post here comparing a Fusion IO card to an 8-disk 15k RPM RAID 1+0 array. Myself I’ve been interested in Fusion IO since I first heard about it, very interesting technology, have not used it personally yet.
The most interesting numbers to me was the comparably poor sequential write performance vs random write performance on the same card. Random write was upwards of 3 times faster.
ESRP. That is what I have started calling it at least. The official designation is Extreme Standby Router Protocol. It’s one of, if not the main reason I prefer Extreme switches at the core of any Layer 3 network. I’ll try to explain why here, because Extreme really doesn’t spend any time promoting this protocol, I’m still pushing them to change that.
I’ve deployed ESRP at two different companies in the past five years ranging from
What are two basic needs of any modern network?
Traditionally these are handled by separate protocols that are completely oblivious to one another mainly some form of STP/RSTP and VRRP(or maybe HSRP if your crazy). There have been for a long time interoperability issues with various implementations of STP as well over the years, further complicating the issue because STP often needs to run on every network device for it to work right.
With ESRP life is simpler.
The optimal network configuration for ESRP is very simple, it involves two core switches cross connected to each other(with at least two links), with a number of edge switches, each edge switch has at least one link to each core switch. You can have as few as three switches in your network, or you can have several hundred(as many as you can connect to your core switches max today I think is say 760 switches using high density 1GbE ports on a Black Diamond 8900, plus 8x1Gbps ports for cross connect).
ESRP Mesh Network Design
ESRP uses a concept of domains to scale itself. A single switch is master of a particular domain which can include any number of vlans up to 3000. Health packets are sent for the domain itself, rather than the individual vlans dramatically simplifying things and making them more scalable simultaneously.
This does mean that if there is a failure in one vlan, all of the vlans for that domain will fail over, not that one specific vlan. You can configure multiple domains if you want, I configure my networks with one domain per ESRP instance. Multiple domains can come in handy if you want to distribute the load between the core switches. A vlan can be a member of only one ESRP domain(I expect, I haven’t tried to verify).
The way ESRP loop prevention works is the links going to the slave switch are placed in a blocking state, which eliminates the need for downstream protocols and allows you to provide support for even unmanaged switches transparently.
Layer 3 fault tolerance in ESRP operates in two different modes depending on whether or not the downstream switches are Extreme. It assumes by default they are, you can override this behavior on a per-port basis. In an all-Extreme network ESRP uses EDP [Extreme Discovery Protocol](similar to Cisco’s CDP) to inform down stream switches the core has failed over and to flush their forwarding entries for the core switch.
If downstream switches are not Extreme switches, and you decided to leave the core switch in default configuration, it will likely take some time(seconds, minutes) for those switches to expire their forwarding table entries and discover the network has changed.
If you know you have downstream switches that are not Extreme I suggest for best availability to configure the core switches to restart the ports those switches are on. Port restart is a feature of ESRP which will cause the core switch to reset the links of the ports you configure to try to force those switches to flush their forwarding table. This process takes more time than in an Extreme-only network. In my own tests specifically with older Cisco layer 2 switches, with F5 BigIP v9, and Cisco PIX this process takes less than one second(if you have a ping session going and trigger a fail over event to occur rarely is a ping lost).
If you are connecting devices like a load balancer, or a firewall directly to the switch, you typically want to hand off loop prevention to those devices, so that the slave core switch will allow traffic to traverse those specific ports regardless of the state of the network. Host attached mode is an ESRP feature that is enabled on a per-port basis.
ESRP does not protect you from every type of loop in the network, by design it’s intended to prevent a loop from occurring between the edge switch and the two core switches. If someone plugs an edge switch back into itself for example that will cause a loop still.
ESRP integrates with another Extreme specific protocol named ELRP or Extreme Loop Recovery Protocol. Again I know of no other protocol in the industry that is similar, if you do let me know.
What ELRP does is it sends packets out on the various ports you configure and looks for the number of responses. If there is more than it expects it sees that as a loop. There are three modes to ELRP(this is getting a bit off topic but is still related). The simplist mode is one shot mode where you can have ELRP send it’s packets once and report, the second mode is periodic mode where you configure the switch to send packets periodically, I usually use 10 seconds or something, and it will log if there are loops detected(it tells you specifically what ports the loops are originating on).
The third mode is integrated mode, which is how it relates to ESRP. Myself I don’t use integrated mode and suggest you don’t either at least if you follow an architecture that is the same as mine. What integrated mode does is if there is a loop detected it will tell ESRP to fail over, hoping that the standby switch has no such loop. In my setups the entire network is flat, so if there is a loop detected on one core switch, chances are extremely(no pun intended) high that the same loop exists on the other switch. So there’s no point in trying to fail over. But I still configure all of my Extreme switches(both edge and core) with ELRP in periodic mode, so if a loop occurs I can track it down easier.
We will start with this configuration:
The non ESRP portion of this configuration is:
enable sharing 1 grouping 1-4 address-based L3_L4 enable sharing 5 grouping 5-6 address-based L3_L4 enable sharing 7 grouping 7-8 address-based L3_L4 create vlan webservers config webservers tag 3500 config webservers ipaddress 10.60.1.1 255.255.255.0 config webservers add ports 1,5,7 tagged
The ESRP portion of this configuration is:
create esrp esrp-prod config esrp-prod add master webservers config esrp-prod priority 100 config esrp-prod ports mode 1 host enable esrp
The only difference between the master and slave, is to change the priority. From 0-254 higher numbers is higher priority, 255 is reserved for putting the switch in standby state.
It is a simple one liner command to each core switch, extending the example above, say you added a vlan appservers with it’s associated parameters and wanted to protect it, the command is:
config esrp-prod add member appservers
That’s it.
There is only one gotcha that I can think of off hand specific to ESRP. I believe it is a bug, and reported it a couple of years ago(code rev 11.6.3.3 and earlier, current code rev is 12.3.x) I don’t know if it is fixed yet. But if you are using port restart configured ports on your switches, and you add a vlan to your ESRP domain, those links will get restarted(as expected), what is not expected is this causes the network to fail over because for a moment the port weighting kicks in and detects link failure so it forces the switch to a slave state. I think the software could be aware why the ports are going down and not go to a slave state.
Somewhat related, again with port weightings, if you are connecting a new switch to the network, and you happen to connect it to the slave switch first, port weighting will kick in being that the slave switch now has more active ports than the master, and will trigger ESRP to fail over.
The workaround to this, and in general it’s a good practice anyways with ESRP, is to put the slave switch in a standby state when you are doing maintenance on it, this will prevent any unintentional network fail overs from occurring while your messing with ports/vlans etc. You can do this by setting the ESRP priority to 255. Just remember to put it back to a normal priority after you are done. Even in a standby state, if you have ports that are in host attached mode(again e.g. firewalls or load balancers) those ports are not impacted by any state changes in ESRP.
Switches:
Cross connect the X650 switches to each other using 2x10GbE links with CAT6A UTP cable. Connect each of the edge switches to each of the core switches with CAT5e/CAT6/CAT6a UTP cable. Since we are working at 10Gbps speeds there is no link aggregation/trunking needed for the edge(there is still aggregation used between the core switches) simplifying configuration even further
Is a thousand ports not enough? Break out the 512Gbps stacking for the X650 and add another pair of X650s, your configuration changes to include:
Two thousand ports not enough, really? You can go further though the stacking interconnect performance drops in half, add another pair of X650s and your configuration changes to include:
The maximum number of switches in an X650 stack is eight. My personal preference is with this sort of setup don’t go beyond three. There’s only so much horsepower to do all of the routing and stuff and when your talking about having more than three thousand ports connected to them, I just feel more comfortable that you have a bigger switch if you go beyond that.
Take a look at the Black Diamond 8900 series switch modules on the 8800 series chassis. It is a more traditional core switch that is chassis based. The 8900 series modules are new, providing high density 10GbE and even high density 1GbE(96 ports per slot). It does not support 10GbaseT at the moment, but I’m sure that support isn’t far off. It does offer a 24-port 10GbE line card with SFP+ ports(there is a SFP+ variant of the X650 as well). I believe the 512Gbps stacking between a pair of X650s is faster than the backplane interconnect on the Black Diamond 8900 which is between 80-128Gbps per slot depending on the size of the chassis(this performance expected to double in 2010). While the backplane is not as fast, the CPUs are much faster, and there is a lot more memory, to do routing/management tasks than is available on the X650.
The upgrade process for going from an X650-based stack to a Black Diamond based infrastructure is fairly straight forward. They run the same operating system, they have the same configuration files. You can take down your slave ESRP switch, copy the configuration to the Black Diamond, re-establish all of the links and then repeat the process with the master ESRP switch. You can do this all with approximately one second of combined downtime.
So I hope, in part with this posting you can see what draws me to the Extreme portfolio of products. It’s not just the hardware, or the lower cost, but the unique software components that tie it together. In fact as far as I know Extreme doesn’t even make their own network chipsets anymore. I think the last one was in the Black Diamond 10808 released in 2003, which is a high end FPGA-based architecture(they call it programable ASICs, I suspect that means high end FPGA but not certain). They primarily(if not exclusively) use Broadcom chipsets now. They’ve used Broadcom in their Summit series for many years, but their decision to stop making their own chips is interesting in that it does lower their costs quite a bit. And their software is modular enough to be able to adapt to many configurations (e.g. their Black Diamond 10808 uses dual processor Pentium III CPUs, the Summit X450 series uses ARM-based CPUs I think)
One of the bigger barriers to adoption of new equipment often revolves around user interface. If people have to adapt to something radically different some of them naturally will resist. In the networking world, switches in particular Extreme Networks has been brave enough to go against the grain, toss out the legacy UI and start from scratch(they did this more than a decade ago). While most other companies out there tried to make their systems look/feel like Cisco for somewhat obvious reasons.
Anyways I’ve always though highly of them for doing that, don’t do what everyone else is doing just because they are doing it that way, do it better(if you can). I think they have accomplished that. Their configuration is almost readable in plain english, the top level commands are somewhat similar to 3PAR in some respects:
Want to add a vlan ? create vlan Want to configure that vlan? configure vlan (or config vlan for short, or config <vlan name> for shorter). Want to turn on sFlow? enable sflow. You get the idea. There are of course many other commands but the bulk of your work is spent with these. You can actually login to an Extreme XOS-based switch that is on the internet, instructions are here. It seems to be a terminal server and you connect on the serial port as you can do things like reboot the switch and wipe out the configuration and you don’t lose connectivity or anything. If you want a more advanced online lab they have them, but they are not freely accessible.
Anyways back on topic, legacy cli. I first heard rumors of this about five years ago when I was looking at getting(and eventually did) a pair of Black Diamond 10808 switches which at the time was the first and only switch that ran Extremeware XOS. Something interesting I learned recently which I had no idea was the case was that Extremeware XOS is entirely XML based. I knew the configuration file was XML based, but they take it even further than that, commands issued on the CLI are translated into XML objects and submitted to the system transparently. Which I thought was pretty cool.
About three years ago I asked them about it again and the legacy cli project had been shelved they said due to lack of customer interest. But now it’s back, and it’s available.
Now really back on topic. The reason for this legacy cli is so that people that are used to using the 30+ year old broken UI that others like Cisco use can use something similar on Extreme if they really want to. At least it should smooth out a migration to the more modern UI and concepts associated with Extremeware XOS(and Extremeware before it), an operating system that was built from the ground up with layer 3 services in mind(and the UI experience shows it). XOS was also built from the ground up(First released to production in December 2003) to support IPv6 as well. I’m not a fan of IPv6 myself but that’s another blog entry.
It’s not complete yet, right now it’s limited to most of the layer 2 functions of the switch, layer 3 stuff is not implimented at this point. I don’t know if it will be implimented I suppose it depends on customer feedback. But anyways if you have a hard time adjusting to a more modern world, this is available for use. The user guide is here.
If you are like me and like reading technical docs, I highly reccomend the Extremware XOS Concepts Guide. There’s so much cool stuff in there I don’t know where to begin, and it’s organized so well! They really did an outstanding job on their docs.
I keep forgetting to post about this, I find this number interesting myself. It is the number of mini RAID arrays on my storage system, which has 200 spindles, which comes out to about 400 RAID arrays per disk! Why so many? Well it allows for maximum distribution of storage space and I/O across the system as well as massively parallel RAID rebuilds as every disk in the system will participate when a disk fails, which leads to faster rebuild times and much better service times during rebuild.
While 3PAR talks a lot about their mini RAID arrays(composed of virtual 256MB disks) it turns out there really isn’t an easy way to query how many there are, I suppose because they expect it to be so abstracted that you should not care. But I like to poke around if you haven’t noticed already!
The little script to determine this number is:
#!/bin/bash
export ARRAYS_TOTAL=0
export ARRAY="mrt"
echo "showld -d" | ssh $ARRAY | grep cage | while read line;
do
export RAWSIZE=`echo $line | awk '{print $7}'`;
export LD_NAME=`echo $line | awk '{print $2}'`;
export SETSIZE=`echo $line | awk '{print $10}'`;
export ARRAYS=`echo "${RAWSIZE}/256/${SETSIZE}" | bc`;
export ALL_ARRAYS=`echo "${ARRAYS_TOTAL}+${ARRAYS}" | bc `;
export ARRAYS_TOTAL="$ALL_ARRAYS"; echo "NAME:${LD_NAME} Raw Size:${RAWSIZE} Set Size:${SETSIZE} Micro arrays in LD:${ARRAYS} Total Micro arrays so far:${ALL_ARRAYS}";
done
Hopefully my math is right..
Output looks like:
NAME:log2.0 Raw Size:40960 Set Size:2 Micro arrays in LD:80 Total Micro arrays so far:80 NAME:log3.0 Raw Size:40960 Set Size:2 Micro arrays in LD:80 Total Micro arrays so far:160 NAME:pdsld0.0 Raw Size:49152 Set Size:2 Micro arrays in LD:96 Total Micro arrays so far:256 [..] NAME:tp-7-sd-0.242 Raw Size:19968 Set Size:6 Micro arrays in LD:13 Total Micro arrays so far:81351 NAME:tp-7-sd-0.243 Raw Size:19968 Set Size:6 Micro arrays in LD:13 Total Micro arrays so far:81364
Like the mini RAID arrays the logical disks (the command above is showld or show logical disks) are created/maintained/deleted automatically by the system, another layer of abstraction that you really never need to concern yourself with.
The exact number is 81,364 which is up from about 79,000 in June of this year. To me at least it’s a pretty amazing number when I think about it, 80,000+ little arrays working in parallel, how does the system keep track of it all?
3PAR isn’t unique in this technology though I think maybe they were first. I believe Compellent has something similar, and Xiotech constructs RAID arrays at the disk head level, which I find very interesting, I didn’t know it was possible to “target” areas of the disk as specifically as the head. I think of these three implimentations though the 3PAR one is the most advanced because it’s implimented in hardware(Compellent is software), and it’s much more granular(400 per disk in this example, Xiotech would have up to 8 per disk I think).
The disks are not full yet either, they are running at about ~87% of capacity, so maybe room for another 10,000 RAID arrays on them or something..
I learned pretty quick there’s a lot more to storage than just the number/type of disks..
(Filed under virtualization as well since this is a virtualized storage post)
..even I was skeptical, though I knew with support it probably wouldn’t be a big deal, a disk fails and it gets replaced in a few hours. When we were looking to do a storage refresh last year I was proposing going entirely SATA for our main storage array because we had a large amount of inactive data, stuff we write to disk and then read back that same day then keep it around for a month or so before deleting it. So in theory it sounded like a good option, we get lots of disks to give us the capacity and those same lots of disks give us enough I/O to do real work too.
I don’t think you can do this with most storage systems, the architecture’s don’t support it nearly as well. To this point the competition was trying to call me out on my SATA solution last year citing reliability and performance reasons. They later backtracked on their statements after I pointed them to some documentation their own storage architects wrote which said the exact opposite.
It’s been just over a year since we had our 3PAR T400 installed with 200x750GB SATA disks, which are Seagate ST3750640NS if you are curious.
Our disks are hit hard, very hard. It’s almost a daily basis that we exceed 90 IOPS/disk at some point during the day, which large I/O sizes this drives the disk’s response time way up, I have another blog entry on that. Fortunately the controller cache is able to absorb that hit. But the point is our disks are not idle, they get slammed 24/7.
Average Service time across all spindles on my T400
How many disk failures have we had in the past year? One? two? three?
Zero.
For SATA drives, even enterprise SATA drives to me this is a shocking number given the load these disks are put under on a daily basis. Why is it zero? I think a good part of it has to do with the advanced design of the 3PAR disk chassis. Something they don’t really talk about outside of their architecture documentation. I think it is quite a unique design in their enterprise S and T-class systems (not available in their E or F-class systems). The biggest advantages these chassis have I believe is two fold:
There are other cool things but my thought is those are the two main ones that drive an improvement in reliability. They have further cool things like fast RAID rebuild which was a big factor in deciding to go with SATA on their system, but even if the RAID rebuilds in 5 seconds that doesn’t make the physical disks more reliable, and this post is specifically about physical disk reliability rather than recovering from failure. But as a note I did measure rebuild rate, and for a fully loaded 750GB disk we can rebuild a degraded RAID array in about three hours, with no impact to array system performance.
My biggest complaint about 3PAR at this point is their stupid naming convention for their PDFs. STUPID! FIX IT! I’ve been complaining off and on for years. But in the grand scheme of things…
Not shocked? Well I don’t know what to say. Even my co-worker who managed our previous storage system is continually amazed that we haven’t had a disk die
Now I’ve jinxed it I’m sure and I’ll get an alert saying a disk has died.
Just thought it was kind of funny timing. Xiotech came to my company a few weeks ago touting their ISE systems, about the raw IOPS they can deliver(apparently they do something special with the SCSI protocol that gets them 25% more IOPS than you can normally get). I asked them about SSD and they knocked it saying it wasn’t reliable enough for them(compared to their self healing ISEs).
Well apparently that wasn’t the case, because it seems they might be using STEC SSDs in the near future according to The Register. What? No Seagate? As you may or may not be aware Xiotech’s unique features come with an extremely tight integration with the disk drives, something they can only achieve by using a single brand, which is Seagate(who helped create the technology and later spun it out into Xiotech). Again, The Register has a great background on Xiotech and their technology.
My own take on their technology is it certainly looks interesting, their Emprise 5000 looks like a great little box as a standalone unit. It scales down extremely well. I’m not as convinced with how well it can scale up with the Emprise 7000 controllers though, they tried to extrapolate SPC-1 numbers from a single ISE 5000 to the same number of drives as a 3PAR T800 which I believe still holds the SPC-1 record at least for spinning disks anyways. Myself I’d like to see them actually test a high end 64-node ISE 7000 system for SPC-1 and show the results.
If your a MS shop you might appreciate Xiotech’s ability to integrate with MS Excel, as a linux user myself I did not of course. I prefer something like perl. Funny that they said their first generation products integrated with perl, but their current ones do not at the moment.
This sort of about face with regards to SSD in such a short time frame of reminds me when NetApp put out a press release touting their de-duplication technology as being the best for customers only to come out a week later and say they are trying to buy Data Domain because they have better de-duplication technnology. I mean I would of expected Xiotech to say something along the lines of “we’re working on it” or something. Perhaps the STEC news was an unintentional leak, or maybe their regional sales staff here was just not informed or something.
10 Gigabit Ethernet has been around for many years, for much of that time it has been for the most part(and with most vendors still is) restricted to more expensive chassis switches. For most of these switches the port density available for 10GbE is quite low as well, often maxing out at less than 10 ports per slot.
Within the past year Extreme Networks launched their X650 series of 1U switches, which currently consists of 3 models:
For those that aren’t into networking too much, 10GbaseT is an ethernet standard that provides 10 Gigabit speeds over standard CAT5e/CAT6/CAT6a cable.
All three of them are line rate, full layer 3 capable, and even have high speed stacking(ranging from 40Gbps to 512Gbps depending on configuration). Really nobody else in the industry has this ability at this time at least among:
The biggest complaints against 10GbaseT have been that it was late to market(first switches appeared somewhat recently), and it is more power hungry. Well fortunately for it the adoption rate of 10GbE has been pretty lackluster over the past few years with few deployments outside of really high end networks because the cost was too prohibitive.
As for the power usage, the earlier 10GbaseT switches did use more power because well it usually requires more power to drive stuff over copper vs fiber. But the second generation X650-24T from Extreme has lowered the power requirements by ~30%(reduction of 200W per switch), making it draw less power than the SFP+ version of the product! All models have an expansion slot on the rear for stacking and additional 10GbE ports. For example if you wanted all copper ports on the front but needed a few optical, you could get an expansion module for the back that provides 8x 10GbE SFP+ ports on the rear. Standard it comes with a module that has 4x1GbE SFP ports and 40Gbps stacking ports.
So what does it really cost? I poked around some sites trying to find some of the “better” fully layer 3 1U switches out there from various vendors to show how cost effective 10GbE can be, at least on a per-gigabit basis it is cheaper than 1GbE is today. This is street pricing, not list pricing, and not “back room” discount pricing. YMMV
| Vendor | Model | Number of ports on the front | Bandwidth for front ports (Full Duplex) | Priced From | Street Price | Cost per Gigabit | Support Costs? |
|---|---|---|---|---|---|---|---|
| Extreme Networks | X650-24t | 24 x 10GbE | 480 Gbps | CDW | $19,755 * | $41.16 | Yes |
| Force10 Networks | S50N | 48 x 1GbE | 96 Gbps | Insight | $5,078 | $52.90 | Yes |
| Extreme Networks | X450e-48p | 48 x 1GbE | 96 Gbps | Dell | $5,479 | $57.07 | Optional |
| Extreme Networks | X450a-48t | 48 x 1GbE | 96 Gbps | Dell | $6,210 | $64.69 | Yes |
| Juniper Networks | EX4200 | 48 x 1GbE | 96 Gbps | CDW | $8,323 | $86.69 | Yes |
| Brocade (Foundry Networks) | NetIron CES 2048C | 48 x 1GbE | 96 Gbps | Pending | Pending | Pending | Yes |
| Cisco Systems | 3750E-48TD | 48 x 1GbE | 96 Gbps | CDW | $13,500 | $140.63 | Yes |
* The Extreme X650 switch by default does not include a power supply(it has two internal power supply bays for AC or DC PSUs). So the price includes the cost of a single AC power supply.
Don’t know what VirtualConnect is? Check this e-book out. Available to the first 2,500 people that register. I just browsed over it myself it seems pretty good.
I am looking forward to using the technology sometime next year(trying to wait for the 12-core Opterons before getting another blade system). Certainly looks really nice on paper, and the price is quite good as well compared to the competition. It was first introduced I believe in 2006 so it’s fairly mature technology.
I was poking around again and came across a new product from Fusion IO which looked really cool. Their new Iodrive Octal, which packs 800,000 IOPS on a single card with 6 Gigabytes/second sustained bandwidth. To put this in perspective, a typical high end 15,000 RPM SAS/Fiber Channel disk drive can do about 250 IOPS. As far as I/O goes this is roughly the same as 3,200 drives. The densest high performance storage I know of is 3PAR who can pack 320 15,000 RPM drives in a rack in their S-class and T-class systems (others can do high density SATA, I’m not personally aware of others that can do high density 15,000 RPM drives for online data processing).
But anyways, in 3PAR’s case that is 10 racks of drives, and three more racks for disk controllers(24 controllers), roughly 25,000 pounds of equipment(performance wise) in the palm of your hand with the Iodrive Octal. Most other storage arrays top out at between 200 and 225 disks per rack.
The Fusion IO solutions aren’t for everyone of course they are targeted mostly at specialized applications with smaller data sets that require massive amounts of I/O. Or those that are able to distribute their applications amongst several systems using their PCIe cards.
Maybe I’m the only one out there, but I am a big fan of Quick Launch in Windows. Any new system I got the first thing I would do is resize the task bar to double height, move the Quick Launch bar to the bottom and start populating it with shortcuts. I hardly ever needed to go to the Start Menu or the desktop to launch programs – in fact, I see using the Start Menu and desktop shortcuts as a complete waste of time.
I tried Windows 7 Beta when it first came out and was very disappointed to see that they eliminated Quick Launch (or I guess some drink-the-koolaid marketing person would say they created a new, better hybrid solution – but whatever, I call it a regression). My one hope was that it was just a feature missing in the beta version and would be back in the release version. Nope. I just upgraded to 7 last night and QL is gone. It really made me want to finally give up and use Ubuntu for my work laptop (already use it for my home laptop). But, since I mostly manage Windows servers at my current job it still is a bit of a pain.
So, what’s that old saying: when Microsoft wants you to drink the koolaid you make lemonade? Something like that.
Here’s what I did to get Quick Launch back:
Done! Now you have a Quick Launch bar that is almost like you had in XP/Vista. Sure, you’re missing the “Add to Quick Launch” feature you used to have in Vista, but at least it’s close. You can quickly add items to your home brewed Quick Launch by doing a right-click-drag from the Start Menu and and choosing Copy here (if you don’t do a right-click-drag, you’ll move it which you might not want to do). You can also right-click and send a shortcut to the desktop, and then drag the shortcut down from there.
Update: After I installed Windows 7 Beta when it first came out I did some searching on this issue and came up with nothing. I haven’t touched 7 since. After posting this article, I did another search and see that others have come up with the same or a slightly different solution. Good! Don’t drink the koolaid!
Powered by WordPress
