Ticket #388 (new bug)

Opened 6 years ago

Last modified 5 years ago

mackie onyx: pitch / samplerate changes during record and playback

Reported by: comotion Assigned to:
Priority: major Milestone: FFADO 2.x
Component: generic Version: FFADO 2.2.1
Keywords: Cc:
The device the bug applies to:

Description

Hello, we have a Mackie Onyx 1640i / FireWire? in our studio which works suprisingly well with jack and ffado. However, we are experiencing intermittent changes in pitch / sample rate which make the mixer both record and output with incorrect pitch and bpm. There are no indications in the jack / ffado logs, even when verbose logs is enabled, that these sample rate events occur. This is a showstopper for us, so we welcome any help to debug this issue and will follow all your advice in obtaining the neccessary information.

Platform is Ubuntu 14.04 64bit 04:00.0 FireWire? (IEEE 1394): Texas Instruments TSB43AB23 IEEE-1394a-2000 Controller (PHY/Link)

Change History

03/01/15 15:17:04 changed by jwoithe

Thanks for your report, which is interesting on a number of fronts. First and foremost, while other users have reported using the 1640i in the past none have - to my memory - reported pitch variations. This by itself doesn't mean a lot because different people use their interfaces in different ways, but it is worth noting.

Are you able to quantify the amount of pitch variation you are seeing, even roughly? For example, are we talking about less than 1%, more than 10%, etc?

How are you clocking the 1640i - from an internal clock or via an external wordclock?

What is the nature of the pitch variation? Does it smoothly shift the pitch, or is it an abrupt change? How long does the erroneous pitch last for?

I don't have any direct experience with the chipset used by the 1640i (my work is in other drivers for other interfaces). However, my understanding of the system employed by the 1640i is that it works pretty much the same as other interfaces in that the sampling clock is derived within the device itself. That is to say that nothing on the computer (eg: FFADO) has anything to do with the generation of the sampling clock. Note that while some firewire audio interfaces can do "sync to firewire" this mode of operation is not supported by FFADO.

What this means is that it is difficult to see how FFADO could be causing a variation in the sample clock of the 1640i. If for some reason FFADO was dropping samples at random the result would be bad distortion (not just a shift of pitch) and in any case this would affect record and playback streams at different times. The only relationship FFADO has with the sample rate of the interface is a command which sends the sample rate to the 1640i which is used once when the streaming is started. This couldn't give rise to the observed problem since it happens during use.

The fact that the pitch shift affects both playback and record confirms that the sample rate of the interface is changing, but FFADO has no control over this when streaming is running. It is therefore worth considering whether there is an external hardware fault at play here. For instance, of the internal OCXO (or whatever is used as the reference timebase in the 1640i) is subject to drift then the problems you observed would be explained by that. Are you in a position to clock the 1640i from an external clock to see if that makes any difference?

About the only way I can see that FFADO could be causing this is if the 1640i for some reason thinks it's in firewire sync mode and it's getting confused by the fact that FFADO isn't sending the relevant information to support this mode. However, the 1640i is not the only interface to use the applicable FFADO driver, and if this were happening (due to an incomplete device setup for example) then I would have expected to see similar reports across a bunch of interfaces. Plus I wouldn't expect things to generally work at all if this were happening - even intermittantly.

In any case, if you are able to fill in the additional information requested above perhaps we'll be able to build up a better picture as what's going on.

03/02/15 10:37:25 changed by comotion

Hello,

On Mon, Mar 2, 2015 at 12:17 AM, FFADO <ffadotrac@ffado.org> wrote:

Thanks for your report, which is interesting on a number of fronts. First and foremost, while other users have reported using the 1640i in the past none have - to my memory - reported pitch variations. This by itself doesn't mean a lot because different people use their interfaces in different ways, but it is worth noting. Are you able to quantify the amount of pitch variation you are seeing, even roughly? For example, are we talking about less than 1%, more than 10%, etc?

About 10%.

This recording has the problem throughout, band says 10% bpm change: https://soundcloud.com/luther-blissett-cabaret/mylder-live-luther-blissett-cabaret-19-februar-2015

This is mixed from 12 channel source material from said mixer in said setup.

So when we record we do not get pitch change, but incorrectly recorded whole recording.

How are you clocking the 1640i - from an internal clock or via an external wordclock?

defaults, so I would guess internal clock.

What is the nature of the pitch variation? Does it smoothly shift the pitch, or is it an abrupt change? How long does the erroneous pitch last for?

Abrupt change. Haven't timed it but the changes typically last minutes up to maybe half an hour.

[snip ffado doesn't generate clock and doesn't sync through firewire]

The setup is in a hacklab so we really appreciate any explanations you can give us about the inner workings of ffado and these firewire devices. This enables us to formulate theories and perhaps do some testing to pin this issue down.

[snip if ffado dropped samples this would distort output ]

Yep, this doesn't sound like xrun or random dropped samples.

playback streams at different times. The only relationship FFADO has with the sample rate of the interface is a command which sends the sample rate to the 1640i which is used once when the streaming is started. This couldn't give rise to the observed problem since it happens during use.

We have tried 48kHz and 44.1kHz rates to see if the device itself might be getting confused but have so far gotten same problems with both rates.

The fact that the pitch shift affects both playback and record confirms that the sample rate of the interface is changing, but FFADO has no control over this when streaming is running. It is therefore worth considering whether there is an external hardware fault at play here. For instance, of the internal OCXO (or whatever is used as the reference timebase in the 1640i) is subject to drift then the problems you observed would be explained by that. Are you in a position to clock the 1640i from an external clock to see if that makes any difference?

I am not aware that the mixer has an external world clock input.

About the only way I can see that FFADO could be causing this is if the 1640i for some reason thinks it's in firewire sync mode and it's getting confused by the fact that FFADO isn't sending the relevant information to support this mode. However, the 1640i is not the only interface to use the applicable FFADO driver, and if this were happening (due to an incomplete device setup for example) then I would have expected to see similar reports across a bunch of interfaces. Plus I wouldn't expect things to generally work at all if this were happening - even intermittantly.

In any case, if you are able to fill in the additional information requested above perhaps we'll be able to build up a better picture as what's going on.

Hopefully I have answered all the questions to satisfaction, and will follow up if y'all need more info.

I also have a Onyx 1220 in my possession which I'm currently checking with newest ffado on arch linux, and will check in the problematic setup as well.

Incidentally, we had a concert with the gear on Saturday, issue happens with jack/ffado stack on debian jessie and ubuntu kx studio.

Regards,

Kacper

03/02/15 14:13:55 changed by nettings

That 10% you report is the difference between 44.1 and 48 kHz. Sounds like the device is switching clocks? Jonathan, at least the Onyx model I used some years ago didn't have an external wordclock in. Can't test anymore, the mixer has since been sold...

(follow-up: ↓ 5 ) 03/02/15 14:59:39 changed by jwoithe

Thanks for the additional information Kacper. As also noted by nettings in comment:3, a 10% variation is the difference between 44k1 and 48. Given also your comment that the change is abrupt it sounds very much like the device is spontaneously switching clocks. I am slightly puzzled that this doesn't create timing-related warnings from FFADO, but there may be additional layers involved within the device which are insulating FFADO from the sync-related effects of the clock change.

It would be interesting to take the recording and find a location where the clock apparently changes. Cut out some audio from after the change and resample it to correct for the change. Then paste it back and see whether this corrects the pitch and speed differences. It will be a little circumstantial but if it sounds reasonably correct after the correction then we have at least made progress.

Related to that, you mentioned that you tried at both 44k1 and 48k. If the problem is due to the device switching between these two rates, then operation at 44k1 would see a slowdown and pitch drop in the fault condition while at 48k it would be a speed up and pitch rise. Is that what you see? Or is it always a change in the same direction regardless of the sample rate used?

Thanks for confirming that the problem occurs on two different machines and software environments. Although I didn't think it was due to the computer itself it's good to have had the opportunity to confirm it.

Regarding word clock it is entirely possible that the mixer doesn't have a word clock input. Many don't. That does of course make it difficult to test clock stability issues. Does the mixer have digital inputs (either ADAT or SPDIF) and do you have something which outputs in this format? Sometimes if a digital input is used the desk will use its clock rather than the internal clock. You'd have to check the documentation to see if this is the case for the 1640i.

Could you explain this comment in more detail:

So when we record we do not get pitch change, but incorrectly recorded whole recording.

Do you mean that when recording you're using the device's outputs for foldback and you're not noticing any pitch change? If the device's sample rate was changing I would expect this - both inputs and outputs would be operating at the same rate at that time so no effect would be evident. The issue is that when the ADC/DAC clock is switching to a different rate (day, from 48k to 44k1) the software is still assuming that the samples being delivered are at the nominal rate (48k). When samples taken at 44k1 are played at 48k there will be an increase in tempo and a rise in the pitch.

It will be interesting to know what your 1220 does. I expect it will be fine.

Oh yes, one more question: there are two variations of the Onyx 1640i: the original (which used an Oxford Semiconductor chipset) and the more recent one which utilises the DICE platform. Do you happen to know which one you are using? The output of "ffado-test Discover" will report the modelid and from that we can determine the variant. Alternatively you could use "-v6" with jackd (after the "-dfirewire" flag) or "ffado-test" to increase the logging level of FFADO and check the copious output to see which driver it's picking.

(in reply to: ↑ 4 ) 03/02/15 15:39:10 changed by comotion

Thanks for your prompt reply Jonatan, we appreciate your help in debugging this issue. I'm going to find answers to your questions tomorrow when I get to the studio, but I would like you to clarify a couple points which will make our tests easier.

It would be interesting to take the recording and find a location where the clock apparently changes. Cut out some audio from after the change and resample it to correct for the change. Then paste it back and see whether this corrects the pitch and speed differences. It will be a little circumstantial but if it sounds reasonably correct after the correction then we have at least made progress.

OK, so we do the following

1) play a note, ie a 440Hz sine wave 2) record it with an external recording device 3) wait for the pitch change 4) take sample after pitch change & resample 5) play back over original sample

The purpose being to validate that the pitch changes ie from 44k1 to 48k.

What program would you recommend for resampling?

Thanks for confirming that the problem occurs on two different machines and software environments. Although I didn't think it was due to the computer itself it's good to have had the opportunity to confirm it.

Again, since ubuntu and debian jessie are so similar we plan on using arch on different hardware to exclude problems with old libraries and kernels.

Regarding word clock it is entirely possible that the mixer doesn't have a word clock input. Many don't. That does of course make it difficult to test clock stability issues. Does the mixer have digital inputs (either ADAT or SPDIF) and do you have something which outputs in this format? Sometimes if a digital input is used the desk will use its clock rather than the internal clock. You'd have to check the documentation to see if this is the case for the 1640i.

This device has the EBU DB25 connector, but we don't have any device to feed it with, except the 1220, and we'll still have to obtain a db25 to db25 cable. If there is an inexpensive ebu device you can recommend for this testing we will certainly try it out.

Could you explain this comment in more detail:

So when we record we do not get pitch change, but incorrectly recorded whole recording.

Do you mean that when recording you're using the device's outputs for foldback and you're not noticing any pitch change? If the device's sample rate was changing I would expect this - both inputs and outputs would be operating at the same rate at that time so no effect would be evident. The issue is that when the ADC/DAC clock is switching to a different rate (day, from 48k to 44k1) the software is still assuming that the samples being delivered are at the nominal rate (48k). When samples taken at 44k1 are played at 48k there will be an increase in tempo and a rise in the pitch.

I do not understand the word "foldback" in this context. You mean monitoring the input during recording? What I mean here is that the entire recording has the pitch change, as opposed to the intermittent abrubt changes that we experience during playback.

It will be interesting to know what your 1220 does. I expect it will be fine.

So far on my arch-based home desktop the onyx 1220 is performing nicely. I experience xruns that my audiofire2 never has in the same setup but I haven't experienced any sample rate problems like on the 1640i in the studio. Will check the 1220 in the studio asap.

I will follow up this issue and answer your questions while we do the testing in the studio, likely tomorrow Tuesday if nothing big gets in the way.

Regards,

Kacper

03/02/15 16:30:22 changed by jwoithe

1) play a note, ie a 440Hz sine wave 2) record it with an external recording device 3) wait for the pitch change 4) take sample after pitch change & resample 5) play back over original sample

Something along these lines would work just as well. In fact it would be easier to get a good fix on the new rate using this than if using music. I suggested the music route because you already have some recorded material and it wasn't clear to me how easy it might be to run other tests. If a test like this is feasible then I would suggest the following variation:

  • Generate a sine wave in software (440Hz as you suggest is fine, but the frequency is in the end arbitrary).
  • Send this to an output of your 1640i at 44k1.
  • Record the 1640i's output using an external device until the fault is heard. Then stop after a few seconds.
  • Repeat the test at 48k sampling.

We can then analyse the recordings to determine the sampling rate in the fault condition. There would be no need to resample in this case because the investigation can be done analytically.

Although not needed if you follow something like the above, for general resampling tasks I tend to use sndfile-resample, which is a command line interface to the excellent libsamplerate; sndfile-resample is supplied with libsamplerate.

This device has the EBU DB25 connector, but we don't have any device to feed it with, except the 1220, and we'll still have to obtain a db25 to db25 cable. If there is an inexpensive ebu device you can recommend for this testing we will certainly try it out.

I am not immediately aware of an inexpensive device with the AES cabling you describe. In any case, using an external digital input like this is only going to tell us something if the 1640i clocks off its digital input. Some do, and some perform internal resampling of the digital input. You would have to check the 1640i documentation to see how it approaches this.

I do not understand the word "foldback" in this context. You mean monitoring the input during recording?

That is what I was referring to, yes.

What I mean here is that the entire recording has the pitch change, as opposed to the intermittent abrubt changes that we experience during playback.

Ah, that is interesting. Does this mean that every recording (whether at 44k1 or 48k) is always at the wrong sample rate from beginning to end? Does the recorded rate vary from run to run or depending on sample rate requested? In light of this I have modified the test I outlined earlier in this post.

Since you mention that the entire recording is made at an incorrect rate, the good news is that you could correct these recordings using libsndfile-resample (or something similar) once we know what rate was used.

So far on my arch-based home desktop the onyx 1220 is performing nicely.

I agree. The xruns are a completely separate issue and can be overlooked for the purposes of fault-finding the 1640i situation.

03/03/15 16:12:02 changed by comotion

So, we have some results after todays testing.

Firstly, this 1640i has the Oxford Semiconductor chipset:

0 0x000ff20500001159 0x00000FF2 0x00001640 Loud Technologies Inc. - Onyx 1640i

Then we have managed to reproduce the problem by performing the test we established.

Links to recordings - directly recorded: Reference recorded by 1220, correct pitch: http://comotion.hackeriet.no/mackie/onyx1220-44k1.wav Pitch problems. recorded on 1640i at 44k1 and 48k: http://comotion.hackeriet.no/mackie/onyx1640i-44k1.wav http://comotion.hackeriet.no/mackie/onyx1640i-48k.wav

Links to recordings - externally recorded: 1640i @ 48k http://lafa.hackeriet.no/recording20150303193236.wav 1640i @ 44k1 http://lafa.hackeriet.no/recording20150303194252.wav 1220 @ 44k1 http://lafa.hackeriet.no/recording20150303200053.wav

Related to that, you mentioned that you tried at both 44k1 and 48k. If the problem is due to the device switching between these two rates, then operation at 44k1 would see a slowdown and pitch drop in the fault condition while at 48k it would be a speed up and pitch rise. Is that what you see? Or is it always a change in the same direction regardless of the sample rate used?

This is indeed what we see, the 48k drops down to 44k1 and the 44k1 recording bumps to 48k. Also, on one of the tests we could clearly see an xrun mark in ardour at the point where the device synced up again to the correct rate.

(follow-up: ↓ 9 ) 03/03/15 18:00:41 changed by jwoithe

Thanks very much for providing these files. I will take a look at them as soon as I can. I should note that I am running audio for a theatre show which bumps in this coming Saturday and has production week all next week. Having received short notice I am still very busy doing preproduction work. As a result there may be a delay of a week or so before I can look at these in detail.

Could you let me know the frequency of the tone that you were playing back through the 1640i during these recordings (this is so I can confirm the 1200 result and get a fix on the relative rate of the recording device)?

the 48k drops down to 44k1 and the 44k1 recording bumps to 48k

Analysis of the recorded files will allow us to confirm this to a high degree of accuracy and is therefore worth doing. When taken at face value the above observation tends to confirm the suspicion that the device is hopping between 44k1 and 48k sample rates during playback.

Could you clarify what happens when recording through the 1640i? From what you've said it simply uses the wrong sample rate for the entire recording (so attempts to record at 44k1 are actually sampled at 48k, and vice versa).

For what it's worth it is really starting to sound as if there is a hardware fault in the 1640i. However, let's get all the facts in before attempting to draw conclusions.

(in reply to: ↑ 8 ) 03/04/15 01:46:08 changed by comotion

Replying to jwoithe:

Thanks very much for providing these files. I will take a look at them as soon as I can. I should note that I am running audio for a theatre show which bumps in this coming Saturday and has production week all next week. Having received short notice I am still very busy doing preproduction work. As a result there may be a delay of a week or so before I can look at these in detail.

That kind of thing happens :-) we are doing a concert on Saturday, but we might use an external recording device.

Could you let me know the frequency of the tone that you were playing back through the 1640i during these recordings (this is so I can confirm the 1200 result and get a fix on the relative rate of the recording device)?

It's not a single tone but rather a guitar beat fed through a delay loop. We had people in the room during testing that would have gone crazy if we had used a single sine note and very nearly did with what we recorded. Will this be sufficient for analysis?

the 48k drops down to 44k1 and the 44k1 recording bumps to 48k

Analysis of the recorded files will allow us to confirm this to a high degree of accuracy and is therefore worth doing. When taken at face value the above observation tends to confirm the suspicion that the device is hopping between 44k1 and 48k sample rates during playback. Could you clarify what happens when recording through the 1640i? From what you've said it simply uses the wrong sample rate for the entire recording (so attempts to record at 44k1 are actually sampled at 48k, and vice versa).

The test verified that when recording as well as during playback on the 1640i there are intermittent changes in pitch and rate. I say intermittent here but during recording the wrong rate was present more often than the correct one.

Since our test was a constant looped sample from the delay pedal, the pitch changes were discernable when playing back the recording over the live input.

For what it's worth it is really starting to sound as if there is a hardware fault in the 1640i. However, let's get all the facts in before attempting to draw conclusions.

Hm, hardware fault or driver/firmware interaction problem, surely?

Regards & best of luck with the theatre show,

Kacper

03/23/15 14:16:33 changed by comotion

Hello, just a quick check to see if you have had a chance to analyse the sound samples? Is there anything else we can do at our end to debug the issue?

03/25/15 03:02:23 changed by jwoithe

Sorry for the silence - as predicted these few weeks are fairly busy for me. Things should improve after this coming weekend.

I haven't had a chance to investigate the audio files but have been thinking more about the situation. In an earlier comment I said "it is really starting to sound as if there is a hardware fault in the 1640i", to which you said

Hm, hardware fault or driver/firmware interaction problem, surely?

Based on what I know to date my suspicion lies with the hardware. Unless there is something very odd going on it's hard to see how any software on the PC could be causing the device to switch sample rates like this while audio streaming is active. Further to this, I believe that others have used this particular device successfully which if true suggests the software side of things is ok.

One thing you could do is to test your unit on a different computer running an OS which is officially supported by the vendor. If under identical test conditions you do not see the clock switching problem then that tends to rule out the interface's hardware. On the other hand, if the problems persist then one must seriously consider the possibility of the interface being faulty.

One thing I just thought of: how is the 1640i powered? Does it have an internal power supply or is it supplied DC from an external plug pack or inline power supply? If it's an external power supply it would be worth using a multimeter and CRO to verify that the power being supplied to the unit under load is good. Bad quality power can give rise to strange effects in digital electronics.

03/27/15 04:49:14 changed by comotion

Hello and thanks for your reply. These are busy times for us all and I appreciate you taking the time to follow up this case.

Before we discovered this problem there have been visitors who have brought in Windows machines and used the 1640i for recording without a hitch. The previous owner of the equipment used OSX to interact with the 1640i without reporting any problem. We will test the equipment again with this setup once we get ahold of similar platforms (win/mac laptops) to verify that nothing has happened with the hardware since we discovered the problem - in effect, since we started using it with ffado/jack.

There is another possibility - that the PC we are using has a broken software/hw stack - perhaps that the firewire card is faulty in some way? We will also try a different computer to interface with the 1640i.

Re: your question about power:

  • the 1640i is a top-of-rack unit and has an internal power supply. However it is entirely possible that the building's AC power is dirtier than usual. We will set up monitoring of this power with an oscilloscope, however this being so timing sensitive it will be tricky to discover if there are any anomalies at the exact moment the sample rate glitches.

Will followup with more hw/software test combos. Any other avenues of investigation we can follow?

03/29/15 04:21:01 changed by jwoithe

Thanks for the updated information. The fact that the interface worked fine under other systems in the past is interesting and certainly seems to reduce the chances of it being hardware related. However, as I've explained earlier there is no obvious way that anything on the software side (that is, FFADO) could be causing the device to randomly switch clock rates.

If the firewire card had a really weird fault which caused it to send old async data to the Onyx from time to time then perhaps this could explain things. That would have to be a very strange hardware fault though. In any case, testing with another firewire card would eliminate that option.

I doubt the mains power would be bad enough to upset the regulation of the DC rails of an internal power supply. If the power supply itself was going bad and dropping out of regulation then this may explain things, but in this case we're back to considering an internal hardware error.

I will continue pondering the situation and await the outcome of the additional tests you mentioned.