Phones Deep Dive

By Giles Moss on June 3rd

When we started this blog, we always intended to spend some time on deep dives into some of our individual systems. Last week we teased you a little over our box office software - and there’s much more to say on that - but today we’re jumping off the diving board with both feet as we take a look into the intricacies of our phone system.

If you observe our busy venue box offices you’ll see the staff spend their whole shifts talking. They’re not chatterboxes (well, maybe some are) but their job is primarily about customer interaction in whatever form presents itself. They reply to emails, talk in person to walk-ins and also answer questions and accept ticket reservations by phone. In fact, they spend a lot of time talking on the phone. And that’s just fine with us!

The history

When we started in Venue 45 we had a phone line from BT but it rarely rung. Customers would walk in, buy a ticket, and that was it. Over the years phone usage has increased dramatically and nowadays a lot of customers like to speak in person to our box offices. Perhaps this will wane over the coming years as people move more towards apps for ticketing, it’ll be interesting to see how things change, but irrespective of what the future holds, for now we need to provide a phone in every box office.

So how to do this? We looked at a few options.

In our church hall we have a landline, that’s easy enough, it worked OK. Our expansion since 2008 has been into hotels and conference facilities which pose a different challenge. The first couple of hotels were asked to provide us with a landline, and although they eventually did, it was clearly a strange request for them and it took a lot of discussion. We decided this wasn’t a good way to go.

We considered getting our own landlines installed, but this was expensive, would have required a complicated negotiation with every hotel and conference venue and would have needed us to rent it all year despite only needing the service in July and August. When we were investigating this option we weren’t of the scale we could have approached a telco for a package of landlines across the city (which is how other venues at the time did it) so it was not an option for us.
While mobile phone coverage in central Edinburgh has improved significantly over the last few years, in 2008 it generally fell apart when the hoards descended upon the city. In fact, it still isn’t completely reliable in some of our box offices especially those that are deep inside buildings. We do use mobiles at times when we need a quick fix, but even then the staff needs to remember to keep them charged and not lose the handset. It’s not ideal and it’s not befitting a professional box office operation.

At the same time as our phone deliberations, our box office ticketing system was being redesigned into something that needed an Internet connection. Needing to provide Internet in all our venues opened the door to the solution to our telephony question, which turned out to be IP (‘Internet’) telephony. We put a phone on every desk, which uses the Internet to connect to our own PBX – our private phone exchange – which we run in a data centre. You can buy Internet-based PBX services from many suppliers now, but we run our own because we’re geeks and it’s the natural way for us to do it.

Our own PBX

Our phone system uses SIP - the Session Initiation Protocol – it’s the standard for Internet telephony. The phones in the desk are SIP clients and our PBX is Asterisk running the new(ish) PJSIP channel. Asterisk is a geek’s paradise. It’s immensely feature-rich and infinitely customisable. It’s also open-source and free for us to use. Our operations barely scratch the surface of what it can offer: We use it for internal or external dialling, voice mail and some IVRs (those things where you “press 1 for sales”). We’ve written a few scripts to link Asterisk’s call handling with our InfoSPACE customer records system too. Our Asterisk servers run on virtual machines in our data centres. The operating system is Debian (Linux), currently Stretch (i.e. Debian 9), running the Asterisk packages from Buster to get us the latest version.

SIP trunks

To connect our Asterisk PBX with the rest of the world’s phone system (the PSTN – Public Switched Telephone Network) we buy a SIP trunking service from a VoIP supplier. This gives us up to 25 simultaneous calls to/from the public network and secures us the external phone numbers we publicise for our venues and production offices. One of the really cool things about this is that we can buy numbers with the Edinburgh dialling code (0131) yet the PBX handling the calls need not be in the same city. By keeping the subscription active all year we secure the numbers allowing us to publicise them far and wide.

We have a handful of mobile phones hooked into our PBX too. This is a neat service from another VoIP supplier that gives us a regular SIM card, which we put into a standard mobile phone, and connects it to our PBX just like any other internal phone. It’s a bit uncanny picking up a mobile handset and calling a box office using a three-digit internal extension! We give these special mobile phones to our senior geeks-in-charge and use them for box offices in particularly odd or temporary locations. We can also use them as surrogate box office phones should we lose the Internet connection to a venue for an extended period.

Networking concerns

We quickly discovered, the hard way, that SIP over the Internet isn’t always straightforward because of NAT. NAT is a technology commonly used to let many people share the same Internet connection and while it works great for web and email, SIP telephony wasn’t designed to handle it. Various bodges have been designed both at the PBX end and in the Internet routers we used in our venues but nothing ever worked completely reliably. The solution was to use VPNs between out venues and data centres, allowing the SIP traffic to pass without NAT getting in the way. We’ll go into a lot more detail on our network setup in another blog post.

Hardware

On the phone hardware side, we like the IP phones from Mitel (formerly Aastra) because, and it may sound a bit daft, their handsets feel very nice to hold and the buttons are pleasantly rubbery. Somewhat more critically, the Mitel phones have a simple yet effective mechanism to retrieve their configuration from a central server. Having a centrally managed system saves us loads of time as we store the configuration for every phone in a spreadsheet, then hit a button to generate various configuration files necessary and upload them to the server. Changing things on the phones therefore doesn't need us to adjust individual phones, we can update the configuration on the server, reboot the phones (remotely if necessary) and the changes are live.

If you thought this blog post has been insanely geeky so far, then buckle up for this next bit!

Numbering Plans

One of the most fun parts of setting up our own PBX was the free reign it gave us to design our own numbering plan. A numbering plan is the set of rules around phone numbers that controls where calls are sent and take it from us, it’s possible to obsess over the details.

Each desk phone needs a dedicated extension so it can be directly dialled. This unique extension number also ties into a remote pickup function, where any phone can answer a call ringing on any other phone, and a function known as hints, which we use in some places to light red lights on phones when other closely-related phones are in use.

We wanted to keep these extension numbers as short as possible and found a 3-digit extension was optimal. We use the first digit to separate functions (2xx numbers are internal phones, 3xx numbers are various helper functions like test and voicemail) and the trailing two digits to identify individual extensions.

Each venue is allocated a block of 10 numbers to standardise things and to give us a bit of room for expansion. Take, for example, the Surgeons’ Hall:

• 240 – main box office number, ring all box office phones so the first assistant to pick up grabs the call.
• 241 & 242 – numbers for the two individual box office extensions
• 243 – extension in the back office at the cashing up desk
• 244 to 248 – free for future expansion
• 249 – production office extension

Our external 0131 numbers for the box and production offices simply connect to these same numbers within the PBX, without the caller knowing of course

• 0131 510 2384 (box office) – rings 240
• 0131 510 2391 (production office) – rings 249

The blocks of 2xx extensions of other venues are currently:

• 200 – Press office
• 210 – Venue 45
• 220 – Mile
• 230 – Niddry
• 240 – Surgeons Hall
• 250 – North Bridge
• 260 – Lister Building (at the Surgeons’ Hall)
• 270 – Symposium Hall
• 280 – Triplex
• 290 – (Future venue)

We’re good to add one further box office, but then we’re out of numbers and will have to stop! (or redesign the numbering - Ed)

The keen-eyed (if anyone is following this still) may notice that Lister block. That’s a curiosity: Lister is a new building adjacent to the Surgeons’ Hall with its own production office extension (and associated external 0131 number) but it will share a box office with the Surgeons. Because it will just have a single production office extension it could fit within the Surgeons numbering block, but that would make it un-standard and we don’t like things that don’t follow a pattern. By allocating it a full 10-number venue block, we get a standard xx9 extension for the production office (the somewhat catchy 269), and if we do end up running a dedicated box office in the future the numbering plan already caters for those extensions.

In addition to the 2xx block, we’ve assigned other blocks as:

• 2xx – venue & production office extensions
• 3xx – test, engineering and voicemail extensions
• 4xx – direct speed dials for Senior Production Team members
• 5xx – extensions in various offices off-site. This includes the bookings and recruitment offices as well as a couple of the Senior Production Team who have SIP phones in their houses (because, why not).

The numbering plan also needs to handle calls to the public phone network. We use 9 as the prefix for external dialling; the PBX is configured to treat numbers starting with 9 as external calls and it sends them to the SIP trunks for onward connection to the PSTN. It turns out this can cause a bit of confusion for some of our staff at the start of the festival before they get used to it. It appears some people these days aren’t used to office phone systems so while it’s second nature to the grizzled old senior team who have been around plenty in their time, for some youngsters having to “dial 9 for an outside line” is a new experience.

4k numbers

Everyone who works for us during the Festival is assigned a four digit extension in the range 4001 to 4999, known as a “four K” extension. This unique number is used as a speed dial from our phone system and acts to make it easier for the on-site tams to call members of staff. Granted, it only saves punching in a few numbers to the phone keypad, but even that is more convenient. When a call is placed to a 4xxx extension, Asterisk executes a Perl script via the AGI facility that queries our staffing database for the true mobile phone number of the staff member. This number is then dialled via a SIP trunk.

Incoming calls

We hold phone numbers for our companies in our customer records system, InfoSPACE. When a call comes in, we check the caller ID against the numbers we have on record (using another Perl script) and if the system can match it to a known company contact that contact’s name will appear on the phone when it rings. It’s not a huge feature but it does show our venue teams that it’s a company calling instead of a prospective audience member.

Outside the Festival

Outside of the festival, when our box and production offices are boxed up and put in storage, the external number for the box office is answered by a recorded message letting the caller know our festival is finished, and the number for the production office connects to the pre-production manager for the venue. The decision over whether to route a call to the ‘outside festival’ holding function, or to connect it to a venue, is taken via a Perl script Asterisk executes with each call. The script looks up a flat text file containing the dates and times at which our various functions transition on and off site. The Venue box offices, for example, open at 0930 on Thursday of week 0 and calls placed after the opening time are routed to the venue phones automatically. This handy automation, while simple, is one less thing we need to remember to do manually when the box offices open for business. Not that we forgot. Too often.

Closing remarks

Our phone system has grown with us and the precise make up of phones and extensions is never the same year-on-year. If you’re still reading, huge congratulations - this turned out to be an insanely deep dive, but it’s something that we really do geek out on and running theSpaceUK gives us a perfect opportunity to be the masters of our own phone system.