SerialPort exception at high receive data rates

[slibert]

I've been testing the C++ WPILibrary-based code for integrating with the nav IMU over the Serial Port, and am experiencing an exception in the SerialPort:Read(char *buffer, INT32 count) function. The CRio is running firmware v. 52, and the WPI Library was installed from WorkbenchUpdate rev 3876 (1/1/2014).

The IMU is configured to transmit packets with a length of about 35 bytes each. The serial port configuration is 57600, 8, N, 1. Flow control is kNone.

The exception occurs somewhere inside viBufRead() and appears to be some sort of buffer overrun condition.

The exception only occurs when the IMU is configured to transmit packets at a high update rate (100Hz).

The exception can take up to 15 seconds or so after opening the serial port before it occurs.

When the update rate is dropped down to 50Hz, the exception does not occur.

Please let me know if any more information is needed on this.

Thanks,

- scott
Mentor, Team 2465 (Kauaibots)

[Kevin O'Connor]

Hi Scott,

Unfortunately I haven't had a chance to dig in and try and replicate this yet and may not until at least Monday, but I have sent this along to some of the NI developers to see if they have thoughts on what might be going on.

[slibert]

Hi Scott,

Unfortunately I haven't had a chance to dig in and try and replicate this yet and may not until at least Monday, but I have sent this along to some of the NI developers to see if they have thoughts on what might be going on.

Thanks, Kevin. Let me know if you need any more information on this.

[Kevin O'Connor]

Thanks, Kevin. Let me know if you need any more information on this.

What rate are you reading the data at and what do you have the read buffer size set to (alternately can you post code you are using which exhibits the issue)? I can reproduce the issue I believe you are describing if I read the data slower than it comes in but ran a simple robot program just reading serial data in a loop for over an hour with the same data rate described without issue.

[slibert]

What rate are you reading the data at and what do you have the read buffer size set to (alternately can you post code you are using which exhibits the issue)? I can reproduce the issue I believe you are describing if I read the data slower than it comes in but ran a simple robot program just reading serial data in a loop for over an hour with the same data rate described without issue.

A Project which exhibits the problem can be downloaded in this .zip file.

https://nav6.googlecode.com/svn/trunk/nav6.zip

A WindRiver project can be found within at crio/c++/nav6SimpleRobotExample

To reproduce the issue, I changed the line in IMURobot.cpp's RobotInit() method from:

uint8_t update_rate_hz = 50;
imu = new IMUAdvanced(serial_port,update_rate_hz);

to:

uint8_t update_rate_hz = 100;
imu = new IMUAdvanced(serial_port,update_rate_hz);

See IMUAdvanced.cpp for the code that intializes/accesses the serial port for buffer sizes, etc.

[Kevin O'Connor]

I haven't been able to cause any errors sending arbitrary data of the appropriate size to that code or to standalone code with the same settings on the serial port. Tomorrow I'll see if I have some time to put together the necessary sample packets to properly emulate the nav6 device to properly explore the various codepaths and see if that reproduces the issue.

Having said that, I will warn that at this point I would be planning on using the settings that currently work for you. If there is a bug and it is inside the VISA code, the ability of anyone outside NI to debug it will be limited and the timetable for getting a fix may be on the long side depending on the severity of the bug and the existing patch timetables for LabVIEW RT. There is a bit of presumption on my part there, I can't speak for NI, but I don't want you to get stuck assuming something will be fixed and then be scrambling if it doesn't.

[slibert]

I haven't been able to cause any errors sending arbitrary data of the appropriate size to that code or to standalone code with the same settings on the serial port. Tomorrow I'll see if I have some time to put together the necessary sample packets to properly emulate the nav6 device to properly explore the various codepaths and see if that reproduces the issue.

Having said that, I will warn that at this point I would be planning on using the settings that currently work for you. If there is a bug and it is inside the VISA code, the ability of anyone outside NI to debug it will be limited and the timetable for getting a fix may be on the long side depending on the severity of the bug and the existing patch timetables for LabVIEW RT. There is a bit of presumption on my part there, I can't speak for NI, but I don't want you to get stuck assuming something will be fixed and then be scrambling if it doesn't.

Thanks, Kevin.

See below for strings representing the two types of messages the nav6 will send:

private static final String ypr_update_message = "!y-000.10 006.64 001.92 352.11C8\r\n";
private static final String raw_update_message = "!qEBDE0624EBA238E7FE5EDD522B02FF1DFFF8FE77 031.846B\r\n";

These are sent at the user-specified update rate. In my test application, the problem appeared at 100Hz. So the raw update message is 53 bytes; at 9-bits per byte (at 8/n/1) this 47.7Kbits/sec, close to the 57600 configured serial bitrate.

However, I've seen the problem at 100Hz even on the shorter "ypr update" messages.

I understand what you're saying about the ability to fix this. I'm encouraging people to reduce update rate if there's a problem. My biggest concern is if other portions of their robot software are taking significant CPU resources, that this might exacerbate the problem and it'd show up even at lower update rates. I'm not sure what the thread priorities at the VISA layer are.

If we don't get a resolution soon, if you have any other thoughts/advice on workarounds, that'd be greatly appreciated.

Thanks!

- scott

[Kevin O'Connor]

Scott, thanks for sample packets. Now that I have the right packets for the code it appears that the serial terminal program I was using to simulate the data does not perform at all as advertised. It looks like it was actually sending ~10 packets per second or less instead of the 100, nowhere near the 10ms update rate it claimed to be set for. I'll try to track down another program to do what I need or grab another cRIO to get Windows out of my way and try that.

[slibert]

Scott, thanks for sample packets. Now that I have the right packets for the code it appears that the serial terminal program I was using to simulate the data does not perform at all as advertised. It looks like it was actually sending ~10 packets per second or less instead of the 100, nowhere near the 10ms update rate it claimed to be set for. I'll try to track down another program to do what I need or grab another cRIO to get Windows out of my way and try that.

Thanks, Kevin.

If that doesn't work for you, I can mail you a nav6 and the power/serial cables to test with.

[kennetho]

So we are running at a lower baud a rate and a lower data generation rate, but are seeing the VI_ERRO_IO error on "viBufRead" on some occasions. Our Serial device is generating 20 characters every 1/10 of a second and sending it at 19200 baud. Once our CRIO starts running without errors, no errors will be seen and the Serial link will run for 10 minutes with no errors detected. If the CRIO is then rebooted, we then may see this error and once seen we get a continuous stream of these errors.

So for us this problem appears to be either a initialization problem with the serial port buffers or the serial port state. We added a reset to the serial port before we started using it and that made the error stop occurring, but when we integrated the code with the rest of our robot code (instead of in a standalone serial test program) we started seeing the error again.

Ken Ocheltree
Mentor with Obots 4122