Recently, we heard from our users that email replies were not being sent to the wall. Further investigation led us to find 15 emailed replies that had not made their way to the wall. Also, several emails were not being read and posted completely (it looked like the system was eating part of the reply!).
The email reply system is now fixed, and here is what we did to fix it.
- First, we manually ran the cron page. This immediately sent out 11 of the 15 unsent emails. Some of these emails were partially eaten. Great, now we realized we had two problems to solve: first why didn't it check the emailed replies earlier, and second why was it eating parts of replies?
- We got our system to output tonnes of debug info on the cron page, and solved why the 4 were not being sent out. Three of them were logical issues (people replying from emails the system did not link to the project). The fourth was most interesting since it lead to solving our second problem. The fourth email was trying to post a blank entry!
- More debug messages later, we found out why the system was eating parts of emails. Our system runs some logic that intelligently (or not so intelligently in these cases) figures out from a reply what part of the email is the actual reply and what part is email thread junk ("the stuff at the bottom of the email, i.e. "In reply to, etc."). Two of these filters were too open ended and when html was put in the reply, it would throw our logic filter. The solution was to remove these two filters so now checking is a bit stricter. A side effect of this is that some replies (in rare cases) may also throw up the 'in reply to' part of the email. But the system will not eat your emails so easily again! One of two problems solved.
- The first problem remains: why did the system not send the 11 replies earlier? Why did we have to call the page manually? Our suspicion lead us to check the cron job that sends reminders. We ran the cron job from the linux shell and it returned an error: "Could not find /o/v/index.php". Aha! We had organized our folder structure in late November. That index.php file was now somewhere else. We put the new location and ran the reminder cron. No errors. Perhaps this is the same issue with our cron that checks emailed replies? We first ran that cron, and sure enough we got the file not found error. Nice! We edited the cron with the same change and ran it again. No errors. Perfection! The cron is now configured to run correctly again and it has been tested. Solved two of two.
We apologize to the authors of those fifteen emailed replies and reassure you that we know why that fail happened and that we have fixed that issue. Let us know if something like that happens, we are very quick to solve these things as they crop up (and we are humble enough to admit that they will crop up here and again despite our intense quality controls.).