The Long Way Around the Phone Company: The Story of our 120 Mile Bandwidth Bypass

Scottsbluff is a community of about 25,000 situated in the North Platte Valley of the Nebraska Panhandle. By most standards, Scottsbluff would be considered little more than a rural outpost. But in many ways, this little town on the prairie is right at the forefront of a breakthrough in the digital divide.

Do It Yourself DSL
In 1998, Scottsbluff was the third town (after Lincoln and Omaha) in the state of Nebraska to have publicly available DSL service – “homebrew” DSL delivered across “dry loops” intended for alarm circuits. My ISP business, Scottsbluff Networx, installed PairGain DSL modems to our first clients that summer and before too long we had several business customers eagerly jumping onto our $200/month 768K service, the first under $1000/month broadband offering in the area. This situation was a win/win for us – our cost for the alarm loops was $8 to $15 per month, and the remaining revenue from the DSL customers allowed us to install more backbone capacity – and our customers were getting a service at a price several hundred dollars a month below the T1 service then available from the ILEC.

The situation did not last long. The telephone company decided to charge a higher rate on the ‘dry loops’ along with much higher installation costs. Our $8/month loops suddenly became $30/month loops, with $150 to $300 line conditioning fees added. Then, in August 1999, we were told by the ILEC that the dry loop service would be discontinued. Some desperate research pointed us in the direction of becoming a CLEC, at which point the ILEC would be required by law to provide us with UNEs – Unbundled Network Elements – which were basically the same circuits that we had been utilizing for the past year. After a couple of months and several thousand dollars of expense, Scottsbluff Networx became a CLEC and was able to continue offering DSL service.

Becoming a CLEC introduced us to a whole new set of problems. To obtain the UNEs, our techs had to undergo training on the archaic and cumbersome line ordering system used by the ILEC. Even after training, the system changed on a regular basis and it took quite a few man-hours to stay up to date on how to order services through it. We were forced to pay large fees for our own CLLI (pronounced silly) codes for each central office that we provided service in. Actually getting the service installed was also a hit or miss proposition, as loops that looked good on the ordering system were often denied once the local tech was sent to do the installation work. Several parts of town were also denied service because of the poor condition of the copper plant. DSL lines would randomly quit working and trouble tickets would go unresolved. The initial promise of DSL service was fading quickly as it became clear that the ILEC would only provide the minimum amount of service required by law while throwing up economic and procedural roadblocks to CLECs. One ILEC required a $30,000 fee just for access to a single central office. When the extra charges and man-hours required to keep up with the regulatory marathon imposed by the ILECs was added up, we were losing money even on $200/month DSL lines. In our area, the deployment of DSL was constrained by political issues and the inability of the ILECs to work with independent ISPs and CLECs. Perfectly good technology was rendered useless by the unnecessary bureaucracy and obstacles put in place by the ILECs.

First Steps into Wireless
Frustrated by the obstacle course laid out by the ILEC with regard to DSL service, we started to explore the possibilities of deploying wireless broadband services. One of the local competitors was already utilizing Cisco 2.4Ghz equipment to provide a service competitive to our DSL. Looking for more information online, I came across the isp-wireless mailing list – a group with several ISPs that were deploying early broadband wireless systems. Marlon Schaefer, the operator of a small ISP in Washington, was one of the early contributors who answered several of my questions and pointed me in the direction of Teletronics, a company that made early 802.11 2 megabits 2.4Ghz access points, amplifiers and antennas suitable for outdoor use.

In February 2000, we deployed our first 802.11 compatible wireless access point. By the end of the year, we had installed seven more access points in Western Nebraska and were providing the first low-cost broadband service to the customers in our service areas. The technology was far from perfect – it was little more than indoor access points and client radios with outdoor antennas attached to them – but it was better than nothing. 20 or so clients seemed to be the limit that we could put on a radio, with about a five mile radius for reliable service. There were also interference issues with other wireless ISPs in a couple of our service areas. It wasn’t until later that I would sort out the mistakes that I was making that were limiting the effectiveness of our wireless system.

In November 2000, I sold my ISP business. The new company owned LMDS spectrum in the Scottsbluff area, and after seeing a demonstration of what the LMDS equipment was capable of doing (multiple T1s of voice and data) it made our Teletronics access points look like baby monitors. Since the LMDS was licensed, it would not be susceptible to interference from outside sources like our 802.11 access points operating in the “junk band” that were vulnerable to microwaves, cordless phones and other access points. At the time, it looked like a good decision to sell out.

The Tech Stock Crash and a Maturing Unlicensed Wireless Industry
While the dot bombs started going off in 2001, the new company continued to grow the ISP business. The dialup customer base reached its peak in spring of 2001 and interest in the DSL and wireless continued to grow. The LMDS plans were delayed as the equipment vendor was acquired by another company and bugs with the client radios were worked out. So far, the LMDS deployment was more hype than results.

The unlicensed wireless system continued to grow and more subscribers were added on a monthly basis. Most businesses still opted for DSL, but several home users started going for the wireless, especially in locations outside of town and out of the reach of cable and dsl. More unlicensed access points were added, simply because they were inexpensive and had an almost immediate return on investment. The system still had a shoddy performance record, but people were buying it anyway. It was still unclear to me whether unlicensed wireless was going to become a legitimate broadband alternative or if it was the next CB radio.

In the spring of 2002, one of the members on the isp-wireless list organized a trade show for wireless Internet Service Providers – WISPCON. I decided to go to this show to get more of a feel for the industry as a whole. When I got to the show, I was quite surprised to find a large and energetic group of “wireless cowboys” that were successfully deploying unlicensed systems all over the world. I already had two years of experience with wireless, which made me an old-timer in the group – but I had a lot to learn. Since I had very little background in RF, I spent a lot of my time in basic sessions describing how the technology works and how to troubleshoot many common problems. More valuable than the sessions were the people that I met at WISPCON. There were wireless pioneers like John Scrivner, Stuart Pierce, Marlon Schaefer and Jeremy Kinsey who were successfully integrating wireless into their ISP businesses and were quick to trade war stories about the good and bad parts of being a WISP. After three days of sessions and social networking with other WISP operators, I came back to Nebraska with a completely different perspective on the industry.

Several things were clear after WISPCON. Almost all of the issues we had been suffering could be easily fixed by applying better engineering to the systems. Utilization of best practices such as cross-polarization, sectorization and utilization of different spectrum for backhaul radios increases the performance and reliability of unlicensed systems substantially. Changing the network infrastructure from bridged to routed and adding bandwidth limit queues increased the capacity of the access points from about 20 customers per radio to 80 customers per radio. Limiting peer-to-peer and other undesirable traffic cleaned up the networks further and we were able to deliver better service to business customers. Learning how many of the RF pros handled equipment installation and even simple items such as learning to weatherproof connectors correctly and the right way to install coax connections were invaluable. All of these changes were relatively easy to apply without spending too much money.

It was also clear that the technology was improving exponentially and that we were going to see amazing price/performance combinations in the very near future. Outdoor adaptations of WiFi systems were starting to drop below the $300 per unit level, making the service much more appealing to residential customers and opening up myriad business plan possibilities. After evaluating the information, I decided to commit all of my resources toward the wireless broadband industry.

Starting Over
In March of 2003, I left my job working for the company that bought my original ISP business. Since my non-compete agreement only covered the state of Nebraska, I partnered up with an existing ISP in Wyoming and set up an entirely wireless ISP in Northern Colorado. I was able to start from scratch and engineer wireless broadband systems without having to deal with older legacy equipment. The results were encouraging. By deploying higher capacity radios in a different piece of unlicensed spectrum the performance of the backbone of the network improved considerably. Instead of being limited to T1 or DSL speeds (normally not more than 1.5 megabits) the wireless network could easily deliver 6 to 8 megabits. Because the backbone radios were operating in different spectrum, the 2.4Ghz spectrum used by inexpensive customer premise radios could be devoted entirely to last mile service delivery. Although total throughput on each sector of the 2.4ghz equipment was limited to about 4.5 Megabits, judicial use of multiple pricing plans (128Kbit up to 2Megabits) allowed us to squeeze more customers on each access point, up to a maximum of 250 on a fully loaded AP.

The Trango radios that we use for our backhaul have several features that make a big difference in the performance and reliability of our network. Since they are multipoint radios, we can install one Trango AP at a strategic location and feed several microcells at distances of up to 20 miles. Instead of being limited to T1 or DSL speeds (normally not more than 1.5 megabits) each microcell can easily deliver 6 to 8 megabits to the customers within range. The Trango also outperforms other 5.8Ghz radios such as the Motorola Canopy, which is only capable of delivering about 1.5meg to each subscriber unit, which is insufficient to feed a heavily populated microcell. Business customers who want higher speeds can be configured to run on the Trango AP units which are capable of delivering lower latency and higher reliability than the standard 2.4Ghz access points that we use for the lower cost residential subscribers.

With the considerable improvements in the wireless industry, the connection to the Internet started to become the bottleneck. In 2001, I was operating a 4000 subscriber ISP on 4 T1 connections, bonded together to form a 6 Megabits connection. With the wireless network, one radio could deliver 8 Megabits so it was clear that more capacity was needed at the backbone side. In Colorado, we were able to work out a favorable deal with a company that had fiber and an OC-12 Internet backbone connection. This particular fiber was hooked to the phone company, but there are several fiber networks around the country that are not owned by ILECs and have a lot of unused capacity. These are ideal places to tie into the Internet backbone, and much more cost effective than the T1 lines that the ILECs normally use to deliver high speed Internet.

Another reason for working with a non ILEC backbone provider is the substantial increase in flexibility gained by not having to make long term commitments. When we needed to upgrade the bandwidth capacity of our Wyoming operation in December of 2002, the only option was to install a DS3 loop and fractional DS3 internet feed. Our order was placed in November of 2002, but service was not fully installed until June 2003, a full four months behind the promised installation date. When we finally started to get the bill for service, the bill was nearly $1400 a month more than what we were quoted. When challenged, the ILEC produced a contract with the higher price that clearly had a forged signature. After several months of trying to come up with resolutions to the late installation, overbilling and invalid contract, we disconnected completely from that company – down to the point of switching to VOIP phone lines at our office. When we disconnected from the ILEC, we were able to replace $10,000 a month worth of services with equivalent services costing only $3200 a month. We are also able to operate under 12 and 24 month contracts for service. ILEC contracts typically have a substantial premium built in for anything less than 36 months. Having the shorter contract times means much more flexibility which is critical in the ISP business. When cheaper rates are available, we are in a position to take advantage of them. This is one of the most valuable lessons I have learned – never trust the telcos and always have a backup plan.

As the months passed and we added subscribers to the system, it proved itself to be reliable and scalable. By spring 2004, the system in Colorado had nearly 500 customers on it, and was hitting 10 megabits of utilization at the peaks. We were maintaining our customer base and even adding customers in areas that had access to cable and DSL service, which was very encouraging. While things in Wyoming and Colorado were going well, developments in Nebraska were about to make things very interesting.

Full Circle
In Nebraska, my original ISP business was sold again and my non-compete agreement was ending in spring 2004. Even though cable and dsl and two other wireless Internet providers were doing business in Scottsbluff, I felt that I could take the lessons learned in Colorado and Wyoming and build an even better wireless network. Finding tower sites didn’t take long and soon we were putting customers on the system. The main problem was the lack of favorable choices for Internet backbone connectivity. The first users were put on using a borrowed T1 connection, but this would only handle the first 100 customers before network slowdowns would cause major problems. A T1 connection from the ILEC was over $1200 per month. The only other network in town was run by a competitor. I had plenty of Internet connectivity, but it was over 100 miles away in Wyoming, and there was no easy way to bring it to Nebraska.

After doing some research, I came across some work done by Travis Johnson in Idaho using Trango point-to-point radios and large dishes to complete a 51 mile link. After going through several potential paths and putting in a lot of driving, I found a tower that looked like it would connect the Wyoming and Nebraska networks. All of the math indicated that the links would work, but it was going to be a complex set of shots involving four radios and covering over 100 miles across a mountain range. I considered several brands of radios for the 60 mile and 40 mile shots, including Orthogon and Redline, but settled on the Trango radios because of previous experience and low cost. The Trango radios would handle 8meg of connectivity and could be easily replaced if more capacity was needed in the future. The two shorter shots (7 miles and 10 miles) would utilize radios with StarOS software on them that delivered 20 megabits capacity at close range.

The equipment was ordered in August and we began setting up the links in September. We were able to get the 40 mile first link from Mitchell, Nebraska to Chugwater, Wyoming operational right away. The shot from Chugwater to Pilot Hill, East of Laramie, was more problematic. It took several days of pointing dishes before were able to get the link operational. When the dish alignment was completed, we ended up with a -63db signal – which is very good for a link of that distance. Testing showed that we were getting 8 megabits of throughput on the link.

Total cost for all of the hardware needed on the 100 mile link was under $15,000. Most of the expense was for the large dishes and the installation hardware. Now that the link is completed, we have our own independent backbone out of the area and we don’t have any dependence on the telephone company or other competitors. We also have the benefit of flexibility and volume pricing on bandwidth shared with the Colorado and Wyoming businesses. Best of all, we don’t have the burden of ungainly long-term contracts with the telephone companies.

There were many easier ways to get our connectivity in Nebraska, but I am very happy with the final result of our efforts. We are able to bypass the phone company completely – even our business phone system is based on Voice Over IP and doesn’t go across the regular telephone system. Customers are thrilled to hear that we have no connection at all to the local phone company, and several businesses are utilizing our wireless service as a backup to their DSL or cable modems in the event of an outage.

What started out four years ago as a desperate experiment has turned into a rapidly growing and economically viable business. I have a three megabit wireless connection at my house that matches the performance and reliability of even the fastest DSL or cable services available in Scottsbluff. Technology on the very near horizon promises to deliver 20 megabits service with only minor upgrades to our system. Even with the current, somewhat limited technology we are using, we are able to successfully compete with DSL and cable providers in our areas and we often take customers from DSL and cable who are fed up with being treated poorly by the bigger corporations.

For a bunch of “baby monitors” this stuff works pretty well.

There is a discussion on one of my wireless ISP mailling lists about how the US is falling behind in broadband deployment and what we need to do to catch up. One of my esteemed colleagues feels that we need to be providing 100meg or GigEthernet to our customers if we are going to be competitive. I think that this is a lofty goal and one that we should all strive toward. However, in light of the current market conditions, I propose that it makes more sense to build networks that are “Just Good Enough” while concentrating on other elements of the business.

While there is a rapid rate of technological change, certain parts of the broadband business do not change at the same rapid pace. Basic business practices such as financial management, maintaining customer relationships, building a customer base, solid billing, good back-end services and developing human resources are more important in many ways than the technology behind the network.

In the Wireless ISP world, there are a few things that do not change as fast as the technology but are still critically important. Obtaining good tower locations, establishing paths for backhaul, developing relationships with tower climbers, knowing tower owners and having an understanding of how basic RF works are a few things that the Wireless ISP needs to do that are all independent of technology used.

Then we have the technology, which changes very rapidly. This is both good and bad. It is great that we have technology that improves at a rapid rate, but it also induces the paralysis of waiting for the “next big thing” when there are plenty of suitable alternatives for use right now that make much more practical and economic sense. One testament to this fact is that one of the most popular BWA (Broadband Wireless Access) platforms is the Alvarion BAII system. The basic technology behind this equipment is ten years old and can only supply a total speed of 3megabits per radio. Even so, there are hundreds of thousands of satisfied customers receiving their high speed internet through these systems. 802.11b WiFi based systems compose an even larger portion of the BWA market, and their adoption has a lot more to do with economies of scale and flexibility of deployment than it does with technology. 802.11a based systems show a tremendous amount of promise, along with their own set of restrictions. It is my belief that the majority of WISP deployments over the next two years will be based on one of these two systems, although there will be major deployments of other proprietary systems such as Canopy, Aperto or Trango. Until WiMAX equipment becomes economically feasible for deployment, this is what the majority of WISP operators will be deploying.

To summarize, if I am asked what kind of network to deploy I would respond with one that is “Just Good Enough” for the current market needs. Develop all of the other things related to the actual service - build the business procedures and customer base & lock up good tower locations while establishing relationships with providers and customers - and start offering service. We are not just building networks - we are building businesses of which networks are a big part. Keeping economic principles and sounds business sense at the forefront of our efforts is far more important than the technology that we use.