Persist can cause store to get into inconsistent state

Created by: imirkin

Let's say that one is using the sql store. Persist looks like this:

func (s *session) persist(seqNum int, msgBytes []byte) error {
        if !s.DisableMessagePersist {
                if err := s.store.SaveMessage(seqNum, msgBytes); err != nil {
                        return err
                }
        }

        return s.store.IncrNextSenderMsgSeqNum()
}

So it will save the message to the messages table, and after that, update the outgoing seqnum in the sessions table.

However if the process is terminated in the middle (or there's a database error), you can end up with the message in the messages table, but the outgoing seqnum will still be the old one in the sessions table. Ideally there'd be some transactionality here, but the interfaces don't have anything like that.

This causes problems when the process is restarted and tries to send a message. It will insert another message into the messages table, which will fail due to a primary key constraint.

I believe a reasonable solution would be to flip those two around. In the failure case outlined, you'll just have burned a seqnum, which seems like a reasonable thing to do in such an exceptional situation.