System Slowdowns and CPU running 100% - Mitel 5000 / MiVoice Office 250

One of the most frequent complaints we get is that of system slowdown. So the user may report that when the phone rings and they lift the handset, that it takes a couple of seconds to answer, or that the display on the phone changes slowly, or when they dial digits they take a few seconds to register.

This is usually caused by the CPU being run at or close to 100%. The avaerage system usually ticks over averaging 20 - 40%, so seeing the processor flat out for a prolonged period rings alarm bells.

Most Common Causes We Have Found

1. Sip Hackers - The number one cause. 90% of issues turn out to be this. All versions of the 5000 have a SIP port 5060 open by default, even the versions that did not support SIP trunks or extensions had a SIP port. If this is open to the internet without a firewall filtering by source IP then your system will get bombarded by call requests, sometimes thousands a second. Even though the calls don't get placed, the effort of responding to them causes the system to slow down.  You can see the messages if you turn on SIP logging. Either lock down SIP using the firewall or turn of the SIP port in IP Settings / advanced
2. SSH Hackers - The same thing will happen if you leave port 22 open to the internet. Some Bot will come along and bombard the system with login messaages trying to find the password. You can see the fail messages in the syslog.  Either lock down or turn off SSH.
3. Voice-mail to email - Whilst it usually does not cause too much of an issue, sometimes the site are heavy voice-mail users and have extended the recording time available per message, or worse yet are using the embedded voice-mail for record-a-call.  The turning of the voice-mail into an mp3 then emailing it can cause the processor to show heavy loading. I tested this once with a 20 minute call recording and it took the processor an hour of running flat out befoire it calmed down.
4. SSL Problems - Getting a bit rarer now, but we have found a site where an SSL error was being shown in the exception log. Using the web interface of the system the certificate was regenerated and the web server restarted. This must have unlocked an internal process as immediately the processor use dropped.
5. ISDN Errors - Whilst Open reach are usually insistent there is no error, we have seen circumstances where ISDN can flood the system with error messages causing the system to slow down as it attempts to handle them all and record them in the logs.
6. Over use of G729 - One of the most common mistakes engineers make which causes load on the CPU and can affect voice quality is to use G729 where it is not necessary. G729 should only be used where there is a real issue with having enough bandwidth for G711. On the LAN where there is bags of bandwidth it is never needed. Even on broadband you may not need it. G729 compresses the speech, which while saving bandwidth, requires more resources on the CPU to do the compression/decompression, so whenever I see a high load on a CPU I look to change to G711 and this usually has a visible effect on CPU use.

I hope this may help someone in the future solve a problem!

John Rogers

Telecom Care Ltd

FreePBX, Asterisk inbound DID restricted to multiple CallerID's

Recently we have been doing a lot of work with FreePBX.  This is an Asterisk distribution and FreePBX is the web interface that writes the Asterisk dialplan.

It was easy to get a DID to ring in to our system, you can do that through the web interface just by filling in the boxes.
Then we were asked to restrict that DID so it only allowed a certain caller to access it.  Again that is fine, there is a caller ID box in the web form that allows you to do that.
Then we were asked if two callers could have access.  This is trickier as the Caller ID box on the inbound route form only allows one number to be entered.  But we created two inbound routes, both with the same DID, but different CallerID's and again this worked, now two phones could dial in on one DID.

Where it started to get unmanageable was when we were asked if there could be 20, 30 or 40 CallerID's with permission to call that DID.  Whilst we could keep creating inbound routes, the menu structure of the web interface was going to start getting very messy showing all those additional routes just for 1 DID.  Because of this we started looking at how to add these into the custom files that are hidden from the web interface.

Here is what we did.

Use WinSCP to connect to the PBX.  WinSCP allows the transfer and editing of files on a linux box.

Navigate  to /etc/asterisk/extensions_additional.conf     -    Don't change anything in here as this is a file FreePBX writes and any changes will be lost.

Open the file and do a search for your DID number, you will see a section like the one below.

[ext-did-0001]
include => ext-did-0001-custom
exten => fax,1,Goto(${CUT(FAX_DEST,^,1)},${CUT(FAX_DEST,^,2)},${CUT(FAX_DEST,^,3)})

exten => 01161234567/07971234567,1,Set(__FROM_DID=${EXTEN})
exten => 01161234567/07971234567,n,Gosub(app-blacklist-check,s,1())
exten => 01161234567/07971234567,n,Set(CDR(did)=${FROM_DID})
exten => 01161234567/07971234567,n,ExecIf($[ "${CALLERID(name)}" = "" ] ?Set(CALLERID(name)=${CALLERID(num)}))
exten => 01161234567/07971234567,n,Set(CHANNEL(musicclass)=default)
exten => 01161234567/07971234567,n,Set(__MOHCLASS=default)
exten => 01161234567/07971234567,n,Set(__CALLINGPRES_SV=${CALLERPRES()})
exten => 01161234567/07971234567,n,Set(CALLERPRES()=allowed_not_screened)
exten => 01161234567/07971234567,n(dest-ext),Goto(disa,1,1)

;--== end of [ext-did-0001] ==--;

What this section of the dialplan does is handle calls in on 01161234567 that have come only from 07971234567.    Notice the include => ext-did-0001-custom .   This causes the PBX to look for another section called [ext-did-0001-custom].   At the moment this section does not exist and we cant put it in this file, but we can put it elsewhere.

Navigate to /etc/asterisk/extensions_custom.conf     -     You may well find this file is empty.

Copy the [ext-did-0001] section and paste it into extensions_custom.conf

Change the section name to be [ext-did-0001-custom] and don't forget to change the section end marker to look the same.  Take out the include statement You can now do a search and replace on the CallerID, it will look like this.

[ext-did-0001-custom]
exten => fax,1,Goto(${CUT(FAX_DEST,^,1)},${CUT(FAX_DEST,^,2)},${CUT(FAX_DEST,^,3)})

exten => 01161234567/07989876543,1,Set(__FROM_DID=${EXTEN})
exten => 01161234567/07989876543,n,Gosub(app-blacklist-check,s,1())
exten => 01161234567/07989876543,n,Set(CDR(did)=${FROM_DID})
exten => 01161234567/07989876543,n,ExecIf($[ "${CALLERID(name)}" = "" ] ?Set(CALLERID(name)=${CALLERID(num)}))
exten => 01161234567/07989876543,n,Set(CHANNEL(musicclass)=default)
exten => 01161234567/07989876543,n,Set(__MOHCLASS=default)
exten => 01161234567/07989876543,n,Set(__CALLINGPRES_SV=${CALLERPRES()})
exten => 01161234567/07989876543,n,Set(CALLERPRES()=allowed_not_screened)
exten => 01161234567/07989876543,n(dest-ext),Goto(disa,1,1)

;--== end of [ext-did-0001-custom] ==--;

You can now save the file and then close it.

If you now ask asterisk to reload the configuration you will get in from two different CallerID's.  You can keep on adding further DID definitions with other CallerID's into this section and they won't show in the web interface of the system.


John Rogers
Telecom Care Ltd

Mitel CSM contacts list format (Callview)

I came across this question today from a customer;

'What is the file format of the contacts import in the CSM server?'

When I looked for the answer it was buried deep in the help file of the CSM Server and not at all obvious.
The file is a six field csv file.  The format is,

Phone number or email, Name, Custom field 1, CF2, CF3, CF4

So a phone number entry might be

01165291911,Bobs Hardware,Bob Hammer,512345

An email contact is defined by an @ at the beginning of the line, so it might look like

@sales@bobshardware.co.uk,Bobs Hardware,Bob Hammer,512345

The fields are matched on calls and emails and the custom fields can be included in reports, so for instance including custom field 2 in the report might result in the customers account number (512345 in the example above) being displayed in a report or used in a routing rule.

The amount of characters allowed per field is set as 255,50,50,50,50,50

John Rogers
Telecom Care Ltd

SMDR (Call Logging) output viewing on the Mitel 3300 (MiVoice Business)

When I looked around the internet for instructions on viewing the smdr output from the 3300's web interface I found a lot of conflicting and disjointed advice.  I thought I would gather it together for my future reference.

I am often in the situation where I am remotely connected to the web interface of the system, but can't telnet, so viewing the real time output is not possible.  But the maintenance command window does allow you to see historic smdr records.

The basic command you could use is -
LOGSYS READ SMDR ALL

This is a bad idea!!!!!!   The system will attempt to output the whole SMDR file onto your screen. The last time I did this there were 20,000 lines of information stored in the file and it took 30 minutes to complete the process.

Usually you will make a test call and then want to see the SMDR output, so you could do this -
LOGSYS READ SMDR NEWEST 20

This will output the last 20 call records and your test call will likely be among them for you to view.

What if I knew the call I was looking for had taken place in the last hour and came from a Nottingham (area code 0115)  number?  In that case I could narrow down the records returned by using the MATCH feature.

LOGSYS READ SMDR NEWEST 200 MATCH 0115

This will take a bit longer to complete as the newest 200 lines have to be searched and just the ones containing 0115 returned to the screen.

Perhaps the examples above will now allow you to make more sense of the section from the manual reproduced below.
LOGSYS READ followed by the qualifiers below

<logname> ALL
<logname> ALL MATCH <string>
<logname> INTO <filename> ALL
<logname> INTO <filename> ALL MATCH X
<logname> NEWEST X
<logname> NEWEST X MATCH <string>
<logname> INTO <filename> NEWEST X
<logname> INTO <filename> NEWEST X MATCH X
<logname> OLDEST X
<logname> INTO <filename> OLDEST X
<logname> INTO <filename> OLDEST X MATCH X

If you know of any other useful lognames that can be inserted above, then please let me know!

John Rogers
Telecom Care Ltd

Mitel Border Gateway tracing

Not being very familiar with linux or the command line I thought I would set down how to do a packet trace on a Mitel Border Gateway as i have been asked to do it on several occasions now.

This captures all the packets going in and out of the interfaces and outputs them to a file which can be read by wireshark.  You are going to need PuTTY (or some other ssh client), WinSCP (or other SCP client) and Wireshark.

Use PuTTY to bring up a command line interface on the MBG and log in as root.  No I cant tell you the root password, it was set by the installer when the MBG was first installed.

At the prompt type   > tcpdump -s0 -i any -w  capturefile.pcap

You can now make calls and keep the capture running until the event or error you are tracking occurs.

To stop the trace type CTRL-C

The file 'capturefile.pcap' will now exist in the root directory

Fire up WinSCP and log into the MBG.  WinSCP provides a file explorer type interface which you can use to get into the root directory and find the capturefile.  Drag and drop it into your harddisk.

The file can now be opened in wireshark for analysis.

John Rogers

Telecom Care Ltd

Loading Xarios Contacts into the Server

If you are using Xarios  and want to add global contact details to the server, there is a function to allow you to do this, but no documentation in the application server manual.

If the site is  on Xarios V2.  This allows you to import a csv file of numbers, names, company names etc.  This is then shown in the call banner when the person calls in.

What it does not allow you to do is see the company contacts in your address book, So you can use it to show who is calling in, but not as an address book for dialling out.

On Xarios V3 the server contacts can be shown in the phone manager address book.

First prepare a csv file.  Each line should have fields comma separated with quote marks around text fields that may have spaces in them.

The first field must be the phone number and the second must be the contact name, further fields (Up to 8) can be added.

e.g.

07989000000,"Scott Millord","Support 4 U"

Next right click on the Xarios server icon in the system tray and select ‘Application Settings’

This opens the web interface, log in as usual

Select Phone Manager / Contacts

The contacts screen allows you to specify the location of your contacts file, and you can either import when you click save, or you can specify a schedule for the server to pick up the info.

You will not see the info on the client unless you change the settings on the client banner.

Right click on the banner and select ‘settings’.

The dialog allows you to add extra fields to the banner, so for instance adding ‘server contact name’ would cause the banner to display ‘Scott Millord’ and then adding ‘server field 3’ would cause the banner to display ‘Scott Millord | Support 4 U’ when the cli was recognised.

Hope this helps!

www.telecomcare.net

What if my database wont load back to my Inter-tel Axxess?

We had a little glitch with a customer this week.  They called in initially to say their voicemail wasn't working, but then decided the whole system had stopped.
After guiding the customer through some diagnostics the engineer went to site.  The cpu (V9.122 CPS) was resetting and not coming back online.
Then engineer was given a backup the customer had done a few days before.  He defaulted the CPU and attempted to upload the database.  The upload failed.  He put the database through the database test and repair tool and tried again. The database would still not upload.
Eventually he upgraded the database to put it onto a bigger CPU he had in the vehicle and got the customer going, even though the system was now unlicensed and would reset every four hours.

On return the original database was tried in a spare CPU and would not upload there either.  We eventually opened the database offline. That worked fine.  We saved it again under another filename and attempted to upload it to the original CPU. That worked.  The CPU went back to site and was put into the customers system.

So the moral is, the database test and repair tool may not correct all errors.  Opening the database offline and then using the database save option is what got it fixed.

Less of these Axxess out in the field now, but still, you may be happy to find this post one day!

John Rogers

Telecom Care Ltd