jack segfault in debugmodule

jackd backtrace

This is weird.

I just tried the same on Debian with ffado trunk, and even without an attached device, nothing fails.

Any chance to try again on i386? Or with an older libc? This somehow reminds me of

https://bugzilla.redhat.com/show_bug.cgi?id=638477

hence I'm asking.

We have duplicate error reports from x86 users as well. However I'll try an older libc a bit later.

As for the glibc memcpy issue I have tested with Linus's patch in LD_PRELOAD and without and without - still segfaults - valgrind out attached here.

I also tried relocating the memcpy functions to memove as described here but also with the same result. https://bugzilla.redhat.com/attachment.cgi?id=487982

Any other suggestions?

Looking at the code, the DebugModuleManager? tries to delete an invalid DebugModul?. So there is a race in the destroy path. Probably concurrent access of the m_debugModules vector....

DebugModuleManager::~DebugModuleManager()
264	{
265	    // cleanin up leftover modules
266	    for ( DebugModuleVectorIterator it = m_debugModules.begin();
267	          it != m_debugModules.end();
268	          ++it )
269	    {
270	        fprintf(stderr,"Cleaning up leftover debug module: %s\n",(*it)->getName().c_str());
271	        m_debugModules.erase( it );
272	        delete *it;
273	    }

Line 272 is the problem.

Hi!

I just copied libffado.so.2 compiled on a Debian system to FC15. Result: works like a charm.

Given that it's the same source, it seems we're talking compiler/toolchain now.

I think the 'delete *it' is wrong because the debug modules register/unregister themselfs at the manager. So the owner of the modules is not the manager.

The manager is a singleton you don't really know when that thing is destroyed. Hence this code depends on the libc order of cleaning things up. For most installation the debug modules have been removed before the destructor from the manager is called. But for few others...

So the simple solution is just to remove the 'delete *it' and everything is fine. In case my reasoning is wrong we leak a few bytes during the shutdown of the system... do we really care?

gdb arfter remove delete

Thanks for looking into this.

Patching as suggested gets us along a little further in the DebugModule? destructor but falls over a line before in the erase.

Can we mitigate this some how in JackDriverLoader?.cpp do you think? ie. get all of our descriptors then unload all of them when done?

Sorry for the late response.

You have to remove the fprintf, since it derefences the pointer too.

Index: debugmodule.cpp
===================================================================
--- debugmodule.cpp     (revision 1979)
+++ debugmodule.cpp     (working copy)
@@ -267,9 +267,7 @@
           it != m_debugModules.end();
           ++it )
     {
-        fprintf(stderr,"Cleaning up leftover debug module: %s\n",(*it)->getName().c_str());
         m_debugModules.erase( it );
-        delete *it;
     }
 
     if (!mb_initialized)

wagi,

I'm afraid to say your suggestions don't work. Even if you remove all code in this loop body, it segfaults at a later stage.

It's so utterly broken that you cannot even debug it, gdb is aborting:

../../gdb/infrun.c:3049: internal-error: handle_inferior_event: Assertion `inf' failed.
A problem internal to GDB has been detected,
further debugging may prove unreliable.
Quit this debugging session? (y or n) y

If we look at the code in question, it's iterating over a vector. As long as the vector is valid, this cannot possibly go wrong. If the vector is invalid, the question is why it's invalid. (memory corruption elsewhere, race condition, whatsoever)

I found a quick workaround: leave everything in debugmodule.cpp as is and simply remove line 61 in ffado.cpp:

// this is very much nescessary, as otherwise the
// message buffer thread doesn't get killed when the
// library is dlclose()'d

static void exitfunc(void) __attribute__((destructor));

static void exitfunc(void)
{
    delete DebugModuleManager::instance();

}

Given that the library compiled on a Debian system runs fine on FC15, I still claim toolchain issue. It's nothing obvious, I can confirm g++-4.6.1 creates working binaries.

I see, I'm firing up my build system again and then I can start to dig a bit deeper.

I think the whole debugmodele concept got a bit out of hand. IIRC, Stefan Richter also complained about ffado not reporting any helpful message if used with jack. I completely agree. I'm trying to figure out how we can simplify things.

Just 2 cents:

Deleting elements from a container while iterating over it is just asking for trouble.

A better solution will probably be:

while( !m_debugModules.empty() )
{
  delete m_debugModules.takeFirst();
}

Just in case this slipped your mind...

Of course, that's normally a very bad idea... I have lost all my C++ skills.

This is the standard way erase&delete is done:

Index: debugmodule.cpp
===================================================================
--- debugmodule.cpp	(revision 1979)
+++ debugmodule.cpp	(working copy)
@@ -264,12 +264,11 @@
 {
     // cleanin up leftover modules
     for ( DebugModuleVectorIterator it = m_debugModules.begin();
-          it != m_debugModules.end();
-          ++it )
+          it != m_debugModules.end(); )
     {
         fprintf(stderr,"Cleaning up leftover debug module: %s\n",(*it)->getName().c_str());
-        m_debugModules.erase( it );
         delete *it;
+        it = m_debugModules.erase( it );
     }
 
     if (!mb_initialized)

Note that I haven't compiled the code nor tested.

Still segfaulting in debugmanager.

I have tried the following. At first the delete line was uncommented and segfaulted the first time there.

I commented that line out and ran into another invalid pointer in DebogModuleManager::unregisterModule which seems to be called from the ~DebugModule? destructor. Commenting out this call gives us a running jack at the expense of some lost bytes. My knowledge of c++ is fairly rudimentry and I can't say I know what's goingn on here. Its interesting to note from jack's output that ~ebugModuleManager is being called twice.

diff -Nur libffado-2.1.0.orig/src/debugmodule/debugmodule.cpp libffado-2.1.0/src/debugmodule/debugmodule.cpp
--- libffado-2.1.0.orig/src/debugmodule/debugmodule.cpp 2009-12-20 02:12:53.000000000 +1000
+++ libffado-2.1.0/src/debugmodule/debugmodule.cpp      2011-04-13 22:56:05.000000000 +1000
@@ -83,10 +83,10 @@
 //              << " at DebugModuleManager"
 //              << endl;
 //     }
-    if ( !DebugModuleManager::instance()->unregisterModule( *this ) ) {
-        cerr << "Could not unregister DebugModule at DebugModuleManager"
-             << endl;
-    }
+    //if ( !DebugModuleManager::instance()->unregisterModule( *this ) ) {
+    //    cerr << "Could not unregister DebugModule at DebugModuleManager"
+    //         << endl;
+    //}
 
 }
 
@@ -263,13 +263,11 @@
 DebugModuleManager::~DebugModuleManager()
 {
     // cleanin up leftover modules
-    for ( DebugModuleVectorIterator it = m_debugModules.begin();
-          it != m_debugModules.end();
-          ++it )
+    while (!m_debugModules.empty() )
     {
-        fprintf(stderr,"Cleaning up leftover debug module: %s\n",(*it)->getName().c_str());
-        m_debugModules.erase( it );
-        delete *it;
+         fprintf(stderr,"Cleaning up leftover debug module: %s\n",m_debugModules.back()->getName().c_str());
+         // delete m_debugModules.back();
+         m_debugModules.pop_back();
     }
 
     if (!mb_initialized)

Success. Well, sort of. With *unmodified* code, that is, current svn HEAD, I was able to build a working libffado.so on FC16.

All I did was to use Debian's binutils, to be precise, version 2.21.0.20110327-3.

I think it's time to escalate the bug to Fedora's binutils managers, at least I'd value their input.

Thanks. Before I hit up the binutils maintainers for comment can you summarise for me what you think is happening? I'm afraid I am out of my depth here.

thanks again, B

Ok. When compiling FFADO under Debian, the resulting library works in both, Debian and Fedora. When compiling FFADO under Fedora, the resulting library neither works in Debian nor Fedora.

So one can assume the code isn't at fault here, but the compiler and its friends create an erroneous binary.

So I replaced Fedora's binutils (containing assemblers, linkers and so on) with the one from Debian, and now Fedora can also create working binaries.

I've narrowed it down to the final linking stage: if I compile everything with Fedora's toolchain and just switch to Debian's ld for creating libffado.so, I get a working binary.

In other words, it seems it's the linker. I cannot draw further conclusions at the moment, it's now up to the ELF experts. If you want to file a bug report, link to this ticket.

I've also attached a small shell script to be run from the source's toplevel directory. That's what I use to overlay the final linker stage with Debian's ld.

shellscript to trigger the final linking stage

Thanks for the info! Will post back any updates here

libffado.so compiled with FC15 toolchain but linked with Debian's ld

libffado.so compiled with FC15 toolchain AND linked with Fedora's ld

JFTR, I've attached two libraries to this ticket, one contains the pure Fedora version (segfaulting) and the other one contains the version which uses Debian's ld (working) for the final linking stage (linking also done on Fedora, just a replaced ld binary and linker scripts).

This might help the binutils guys or ELF experts to look for differences.

Performing the last link step with ld.gold also provides a working libffado.so . This was done with revision 1913 in Fedora 15

static constructors test program

If I sort the symbols by address and compare them, I get, among others, this difference:

diff -u libffado-debian-ld-objdump.txt libffado-fedora-ld-objdump.txt
...
 0025f970 l    d  .eh_frame     00000000              .eh_frame
 002a3c04 l     O .eh_frame     00000000              __FRAME_END__
 002a3c08 l    d  .gcc_except_table     00000000              .gcc_except_table
-002b4000 l     O .ctors        00000000              __CTOR_LIST__
-002b4000 l    d  .ctors        00000000              .ctors
-002b41c0 l     O .ctors        00000000              __CTOR_END__
-002b41c4 l     O .dtors        00000000              __DTOR_LIST__
-002b41c4 l    d  .dtors        00000000              .dtors
+002b4000 l    d  .init_array   00000000              .init_array
+002b41bc l    d  .fini_array   00000000              .fini_array
+002b41c0 l     O .ctors        00000000              __CTOR_LIST__
+002b41c0 l    d  .ctors        00000000              .ctors
+002b41c4 l     O .ctors        00000000              __CTOR_END__
+002b41c8 l     O .dtors        00000000              __DTOR_LIST__
+002b41c8 l    d  .dtors        00000000              .dtors
 002b41cc l     O .dtors        00000000              __DTOR_END__
 002b41d0 l     O .jcr  00000000              __JCR_END__
 002b41d0 l     O .jcr  00000000              __JCR_LIST__

Apparently, the Fedora linker moves most constructor/destructor calls of static objects from the .ctors/.dtors to the .init_array/.fini_array sections, where they are probably not found by the C++ runtime.

Please check whether the attached test program fails with the Fedora linker.

On further thought, the problem is probably not that the constructors aren't executed, but that their execution order changes: GCC bug 46770.

Thanks for tracking this down. I figure from that we should fix our code then. This proves why singletons are just an anti pattern :)

Rebuilding binutils-2.21.51.0.6 with --enable-initfini-array=no also produces a working libffado.so .

Patch to pass link flags into scons

To get around this issue we are going to use the ld.gold linker until a resolution can be found from binutils/upstream.

I've modified the scons scripts as per the patch above to accept an alternative path where we point ld to /usr/bin/ld.gold - is this how you'd recommend doing this or is there a more obvious way? Not sure if you want to include something like this in a similar way to COMPILE_FLAGS.

thanks

Brendan

FYI, I have done some work on removing the singleton debug module. The code is available here:

http://repo.or.cz/w/ffado.git/shortlog/refs/heads/debug-cleanup

This needs some more work because I have removed quite a few debug levels and features, which nobody seemed to use anyway. But it breaks probably some of the debug messages people expect to see. So we need some coordinated review of the debug messages.

Could someone try someone who is seeing this bug the above version?

Thanks

JFTR, it's no longer possible to hide the problem with binutils-gold as of 2.21.53.20110922-1 (Debian version).

However, the backtraces still differ between binutils-gold and plain binutils.

I have recently upgraded my development PC such that it now includes binutils 2.21.x. Previously (on a system with binutils 2.20...) I was not able to reproduce this bug. If my system now exhibits these issues I intend to have a closer look. It would appear that the binutils version thing may have been a false lead and that instead we're dealing with a memory-corruption type bug. The tricky bit is that it might not even be in the debug module - that might just be the thing which happens to be taken out by the underlying problem. Therefore I think it's important to understandthe cause of this issue if at all possible even though the existing debug module will be replaced with Daniel's rewrite at some point.

I've now recompiled trunk (presently r1995) in the new environment (gcc 4.5.2, binutils 2.21.51.0.6.20110118). The crash in the debug module (or anywhere else for that matter) still does not occur on this system. The behaviour does not change regardless of whether a firewire device is plugged in or not. The lack of consistency suggests to me that we might be looking at a memory corruption or initialisation issue somewhere (which may or may not be in the debug module). Not being able to trigger the bug myself does make it a little difficult to investigate. Hmm.

Weird. In contrast to my i386 Debian unstable system, I cannot reproduce this bug on my amd64 Debian unstable system.

OTOH, this ticket contains crashlogs from an amd64 machine, so it's not an amd64<->i386 thing.

Jonathan, if you like, you can have a login on my i386 machine. Just contact me, if interested.

possible fix

I've added ffado-linker.patch to this ticket. Those who like, give it a whirl. It fixes the problem for me, and I can't see no obvious negative impact so far, but broader testing (and subsequent feed-back) would be highly appreciated.

Adi: I'm going to be short of time today and possibly tomorrow but I'll check the effect of the patch out on my box as soon as I can (keeping in mind that to this point I haven't seen the crash). What exactly does the "nodelete" linker switch do? I'll have to look it up.

Even so, what's your feeling as to what this patch is doing - do you think it's fixing the problem, or is it instead working around some other issue? Like you said in comment 31, it's strange that it seems to come and go from system to system, which is why I still wonder about there being some memory corruption happening somewhere in the system. Precisely how this linker option effects any sort of behavioural change is still unclear to me. Time and testing will tell though.

News. "-znodelete" doesn't fix the problem. While it hides the problem when calling libffado2 from jackd, it will still crash with this minimal test program:

echo 'int main() {}' |gcc -x c -lffado - && ./a.out

While debugging, I've noticed that XMLDeserialize calls the DebugModuleManager? *constructor* rather late (I'd say upon termination). This then might or might not collide (race condition) when accessing m_debugModules.

valgrind's first complaint:

Cleaning up leftover debug module: DeviceManager
==14771== Invalid free() / delete / delete[]
==14771==    at 0x4023EAC: operator delete(void*) (in /usr/lib/valgrind/vgpreload_memcheck-x86-linux.so)
==14771==    by 0x41241F1: DebugModule::~DebugModule() (debugmodule.cpp:91)
==14771==    by 0x4124821: DebugModuleManager::~DebugModuleManager() (debugmodule.cpp:294)
==14771==    by 0x4121569: exitfunct (ffado.cpp:61)
==14771==    by 0x400E4C9: _dl_fini (dl-fini.c:244)
==14771==    by 0x42DA4DE: __run_exit_handlers (exit.c:78)
==14771==    by 0x42DA54E: exit (exit.c:100)
==14771==    by 0x42C1E4D: (below main) (libc-start.c:260)
==14771==  Address 0x428941c is 0 bytes inside data symbol "m_debugModule"

(ignore the line numbers for debugmodule.cpp, I've hacked a bit on it)

Anyway, still no useful indication where to go. The best advice I have atm is to drop line 61 in ffado.cpp.

Hmm. I'll be the first to admit that I'm not up to speed on all the exotic features of C++, but something doesn't quite add up to me here. The exitfunct calls the debug module destructor; fair enough. This iterates over the debug modules which are still associated with the manager to get rid of them. This seemingly deletes the iterator before using it in the "++it" operation as part of the "for" loop. Ordinarily this would count as a dereference after deallocation; are the rules different for C++ iterators - in other words, is this a valid use of iterators in C++?

Then there's the issue of erasing the current item from the iterator. Is that valid?

Having written that, I reckon this discussion has come up before. If it has, the continued existence of this code would suggest that it was decided that it was acceptable.

Jonathan,

I tried debugging a little bit today and your remarks seems to be leading in the right direction.
But before some thoughts:

1.) I don't have this error on my machine. To reproduce it, I had to comment out the erase-statement in DebugModuleManager::unregisterModule.
After doing that I always had the error.
The question is: What prevents modules from being unregistered on some setups?
I found out, that - even when commenting out the erase-statement in unregisterModule - the DebugModule? destructor is called. (But I did not find where the Module is deleted)
But it might be a sign that the modules are already deleted and only the reference in the vector is not erased. This would explain why even the getName() is not working...
In this case, erase would be prefectly sufficient in ~DebugModuleManager()

2.) ~DebugModuleManager() is called twice during jack startup!!! This seems weird to me...

Back to the vector:
The increment in the "for loop" is at least unnesessary. Erase seems to position the iterator to the next element. The additional increment will result in deleting only every second element in the vector..., this could cause trouble when the second last element is erased: The iterator is at the last element but is incremented again...

BTW:

In RegisterModule() is a return-statement in the for loop, which makes all the "already_present stuff" useless.

Holger

Replying to jwoithe:
Most changes to a vector invalidate all iterators into this vector. So neither are you allowed to dereference any iterator after calling erase(), nor is it possible to reliably continue the iteration.

Consider a simple vector implementation where its contents are a plain array and the iterator type is a plain pointer. After erasing an element, it will have been overwritten by the next element. Furthermore, after erasing the last element, incrementing the iterator makes it point to the element behind the new end().

Replying to dehnhardt:

Ok, replying to my own comment;-)

1.) I don't have this error on my machine. To reproduce it, I had to comment out the erase-statement in DebugModuleManager::unregisterModule.

I have overseen, that this is called in the destructor of DebugModule.

This explains, why every operation on this module in ~DebugModuleManager() fails.

BUT this seems to reproduce exactly the behavior of this bug. Could it be, that calling unregisterModule may fail in the DebugModule destructor under some circumstances?

To proove or disprove, could someone who suffer from this bug apply the following patch, start and stop jackd (without any ffado debug level) and post the output?

Holger

PATCH

Index: src/debugmodule/debugmodule.cpp
===================================================================
--- src/debugmodule/debugmodule.cpp     (Revision 1998)
+++ src/debugmodule/debugmodule.cpp     (Arbeitskopie)
@@ -83,6 +83,7 @@
 //              << " at DebugModuleManager"
 //              << endl;
 //     }
+    cerr << "*~* delete DebugModule " << this << endl;
     if ( !DebugModuleManager::instance()->unregisterModule( *this ) ) {
         cerr << "Could not unregister DebugModule at DebugModuleManager"
              << endl;
@@ -265,11 +266,12 @@
     // cleanin up leftover modules
     for ( DebugModuleVectorIterator it = m_debugModules.begin();
           it != m_debugModules.end();
-          ++it )
+           )
     {
-        fprintf(stderr,"Cleaning up leftover debug module: %s\n",(*it)->getName().c_str());
+       cerr << "*~* erase from vector DebugModule " << *it << endl;
+        //fprintf(stderr,"Cleaning up leftover debug module: %s\n",todel->getName().c_str());
         m_debugModules.erase( it );
-        delete *it;
+        //delete *it;
     }
 
     if (!mb_initialized)
@@ -445,7 +447,8 @@
     {
         if ( *it == &debugModule ) {
             already_present=true;
-            return true;
+           break;
+            //return true;
         }
     }
 
@@ -464,7 +467,7 @@
 
     for ( DebugModuleVectorIterator it = m_debugModules.begin();
           it != m_debugModules.end();
-          ++it )
+          it++ )
     {
         if ( *it == &debugModule ) {
             m_debugModules.erase( it );

Just to say that this bug is also happening at Ubuntu Oneiric, as described by bug https://bugs.launchpad.net/ffado/+bug/869420

The current version used by Ubuntu is from svn r1985, built with binutils-gold (2.21.53.20110810).

Replying to dehnhardt:

Replying to dehnhardt: Ok, replying to my own comment;-)

1.) I don't have this error on my machine. To reproduce it, I had to comment out the erase-statement in DebugModuleManager::unregisterModule.

I have overseen, that this is called in the destructor of DebugModule. This explains, why every operation on this module in ~DebugModuleManager() fails. BUT this seems to reproduce exactly the behavior of this bug. Could it be, that calling unregisterModule may fail in the DebugModule destructor under some circumstances? To proove or disprove, could someone who suffer from this bug apply the following patch, start and stop jackd (without any ffado debug level) and post the output?

Output from jackd when using libffado with your patch, on Ubuntu (on top of r1985): http://paste.ubuntu.com/704633/

I don't have a lot of time right now (I have to go out), but I am really questioning the validity of doing the erase() call in the way it's currently done in the DebugModuleManager? destructor. Clement's comment certainly seems to confirm my gut feeling that it isn't valid code.

Calling unregisterModule() from the DebugModule? destructor isn't a fundamental problem AFAICT, and in fact would seem to be the right thing to do for other reasons. When the DebugModuleManger? destructor runs it calls erase before deleting the individual DebugModules?, so when the unregisterModule/erase call is called from the DebugModules?() the erase won't be called a second time because by that point the module is no longer in the manager.

More to come when I get back online.

I think what we ought to do is try to clean up these destructors which use iterators to delete objects from lists. The construct under suspicion in the debug module system is used in many other places within FFADO; if it is indeed invalid code then this may go a long way to explaining the random crashes we're still seeing on exit on certain machines.

The typical pattern we're seeing is as follows.

  for (Iterator it = SomeVector.begin(); it != SomeVector.end(); it++) {
    SomeVector.erase(it);
    delete *it;
  }

In other words, this goes through SomeVector? and deletes each element in it. At least that's the intention.

Checking the STL documentation, it seems that the above pattern is wrong - this is consistent with Clement's advice earlier. An iterator erase() method "Removes from the vector container either a single element (position) or a range of elements", and "This invalidates all iterator and references to position (or first) and its subsequent elements".

Since the call to erase() invalidates the iterator, it seems to me that we simply can't use vector iterators to effect cleanup of vectors and their elements. At the very least, after erase() the iterator is invalidated and thus the use of it in the following delete() operation is undefined.

The question is what form this alternative might take. I'm thinking of something like this:

  while (!SomeVector.empty()) {
    SomeVector.pop_back();
  }

The pop_back() method calls the removed object's destructor, so that side of things is taken care of. What I'm not sure of is whether a dynamically allocated object's memory will also be freed by this action. I don't think it is, in which case something like this may be in order:

  while (!SomeVector.empty()) {
    Element e = SomeVector.get_last_element();
    SomeVector.erase_element(e);
    delete e;
  }

The idea here being that we get the last element in the vector, remove it from the vector, then call delete() on it to invoke the destructor and free its memory. Then repeat until the vector is empty.

By way of concrete example, I'm considering something like this in the context of ~DebugModuleManager?() in place of the existing iterator-based deletion of m_debugModules elements:

  while (!m_debugModules.empty()) {
    DebugModule *mod = m_debugModules.back();
    m_debugModules.erase(m_debugModules.begin()+m_debugModules.size()-1);
    delete mod;
  }

I've compile-tested this on my box, but because the error condition is not triggered for me I don't know whether it helps or not.

Note that we can't use m_debugModules.end() as the argument to erase() because end() returns a reference to the "just past the end" element, not the final element itself.

Of course in the light of this whole discussion, it could well be that the debug module segfaults are just the end result of memory corruption caused by one of the other iterator-erase-delete constructs which have executed earlier. Since it seems that the existing construct could do crazy things, any one of these under the correct circumstances could give rise to the problems we're seeing.

If the above suggested construct makes sense to those more familiar with C++ than I, I'll go through and replace all the existing iterator-erase-delete loops with this new version and see if it changes things.

Replying to jwoithe:

... The question is what form this alternative might take. I'm thinking of something like this:

  while (!SomeVector.empty()) {
    SomeVector.pop_back();
  }

The pop_back() method calls the removed object's destructor, so that side of things is taken care of. What I'm not sure of is whether a dynamically allocated object's memory will also be freed by this action.

If the vector's element type is a pointer, it is only the pointer itself that is destructed. If the SomethingManager object owns the Something objects, you might consider using smart pointers.

I'm considering something like this in the context of ~DebugModuleManager() in place of the existing iterator-based deletion of m_debugModules elements:

  while (!m_debugModules.empty()) {
    DebugModule *mod = m_debugModules.back();
    m_debugModules.erase(m_debugModules.begin()+m_debugModules.size()-1);
    delete mod;
  }

This looks correct, but you can use pop_back() instead of erase(...).

As the same code is used multiple times, consider using a helper function:

template<typename T>
void delete_pointers_from_vector(vector<T*>& vec)
{
  while (!vec.empty()) {
    T* ptr = vec.back();
    vec.pop_back();
    delete ptr;
  }
}

Or even more C++ish:

template<typename T>
inline void delete_ptr(T* ptr)
{
  delete ptr;
}

template<typename T>
void delete_pointers_from_vector(vector<T*>& vec)
{
  vector<T*> d;
  d.swap(vec);
  for_each(d.begin(), d.end(), delete_ptr<T>);
}

Replying to cladisch

Thanks for the comments and clarification. I'll do up a patch incorporating this in the next day or so; then we can see whether it makes any difference to those experiencing the crash on exit. Even if it doesn't fix the crashes it seems like it's a fix we need to make regardless.

Fixed in r1999. Kudos to David Henningsson for the patch (via e-mail).

For the record, here is the text of David's post to ffado-devel which explains the rationale of the above patch:

I can reproduce the crash, and have attached a patch which fixes it here.

There are two problems, which both are already mentioned in the report:

First, debug modules are not allocated pointers, in fact, the standard way of making one (using DECLARE_DEBUG_MODULE macro) does not. Therefore you must not delete it.

Second, DebugModuleManager::instance() becomes a dangling pointer after the instance has been deleted, so DebugModule? instances must not reference it.

The attached patch fixes both problems.

In the longer term we should also fix the issue of DebugModuleManager::instance() becoming a dangling pointer on close-down, for completeness. With the revised code though this isn't a current problem. Also, similar uses of iterators to delete member objects of other manager functions in ffado also need to be addressed along the lines of that outlined in comment:42 and comment:43. While most of these occur in destructors they do crop up in other places as well.

Note: the problems discussed here may have been related to those raised in ticket #283.