Can't add any new data or raise support ticket!

Help & advice
1

Hi Team,

I am posting here as I cant get a support ticket through to you - Today we basically cant do anything in Breww other than navigate existing data.

Adding anything new including new sales, products or even support tickets leads to
“403 Forbidden CSRF verification failed. Request aborted”

And occasionally we are getting “Sorry, but there was an error processing your request. Please try again and contact the Breww team if problems continue.” for things such as adding leads or contacts.

If you can please advise when we will be back online as it is causing significant problems for us.

Feel free to email me on ob@mollyrosebrewing.com should you need to get in touch.

Cheers

2

We are experiencing the same issue.

3

Same here. It was working early this morning (brewing since 5:30am AEST), but then started failing at around 7 at a guess. Attempted to log a support ticket, but same error when submitting. I’ve emailed my only contact there (sales based, so not sure it’ll achieve anything). I note their support hours don’t officially start until 6pm AEST… :frowning:

4

Same issue here unfortunately

5

Same issue here, old data yes, but anything from racking, transfers or whatever new, it prompts the aforementioned error 403

6

we also have the same problem

7

Same for us! It seems all form posts are broken due to CSRF issues that prevents browsers from sending data.

8

Us too, would be good to have a service status page for these type of situations @breww-team

9

Hello everyone. I’m really sorry about this problem; this should now be resolved now.

We’re looking into why this happened and will post an update when we’ve had time to review it.

If anyone is still having a problem, please let us know.

10

Agreed - plus some kind of mechanism for alerting the team when the ticket mechanism is also impacted. Now it’s back I’ve raised a ticket to flag this.

11

Appears to be sorted now … bit of a ‘forced to do other things’ kind of day … got some stuff off my to-do list though :wink:

12

Again, I’m really sorry we had this issue. We fully understand the impact it has on our customers and and severity of it. We aim never to have problems like this and fortunately, they are extremely rare. We will learn from this to improve things for the future. I’m pleased you managed to get to some other things on the to-do list :grimacing:

As I said before, we’ll post a more complete update about this soon, but I thought it was worth highlighting to people reading this now that we do already have a status page. Our status page can be found at status.breww.com.

We have extensive monitoring in place for Breww and the alerting of issues detected will wake us up in the night, breaking through our phone’s “do not disturb” settings, always being at full volume, etc. To compound the impact of this particular problem, the “extensive monitoring” didn’t catch this issue. If the monitoring had alerted us to the problem, the issue would have lasted a handful of minutes only. Getting everything back up and running was quick when we were awake and aware of the problem.

In light of this issue, we will, of course, improve the monitoring in the future.

We will be updating the status page soon to acknowledge this incident and will post more information on what happened and why on the status page.

I think the community is probably the best place for this. The community deliberately shares very little with our main infrastructure to ensure that it’s unlikely to be impacted by the same problems. Posting on the community won’t wake us up in the middle of the night, however. And as much as giving our customers a mechanism to do this would have been helpful here, I’m not sure it’s the right thing to do, to be honest. Usually our monitoring will discover problems and ensure we know about them 24 hours a day.

Apologies again to everyone. Unfortunately, issues can and will occur, as they do in businesses of all shapes and sizes. The best we can do is ensure we handle them well. We thought we had excellent processes in place for this, and on the whole, we do, but they were not good enough here. We’ll make sure they’re improved.

Thank you everyone for your patience and understanding :pray:

13

Thanks Luke. Appreciate the feedback.

14

Thanks for all the info Luke.

15

We had the same. Seems back now but it was a bit of a worry moment!

16

Thanks for the update Luke, appreciate the feedback