Have you tried wobbling the speaker wires and connector when it goes out? Could be a soldering problem where wires attach to speaker or where speaker connector attaches to main board. Also make sure the female plug is seated all the way into the male socket.
Less likely, if you have a 4G module, then it could also be a problem with soldering at the surface-mount components that are in the speaker circuit or the solder-jumpers also in that circuit. These components and jumpers are just below 4G module on the main board. There are also some surface-mount FETs up next to the speaker connector on either 4G or 2G boards.