Growing Pains - Maintenance Service full re-write

 9 April 2019

Hi everyone,

You may not be aware but one of the major working components of our system is the "maintenance service". The maintenance service is responsible for most of the background processing of records and sightings to ensure that things show up at the right time and on the right pages after new records are received or when existing records are updated.

Due to our growth, we are now in a situation where the maintenance service cannot keep up with the demand being placed on it. This is pretty much a 3 year old architecture that I originally put in place to alleviate the web site performance issues we were experiencing back then (again due to our growth). The maintenance service did an awesome job and took over processing by moving it offline into background operations so that website users didn't have to wait..

But here we are 2-3 years later, now having maxxed out that architecture and needing to fully rebuild it to handle the next order of magnitude in our growth.

My colleague, Rohan Thomas, has been working extremely flat out day and night to modernise the maintenance service by replacing it with a brand new scalable, queue-based architecture. This is the stuff that powers real enterprise level software systems and the kind of design that we work with day in day out at our normal day jobs. That same design has now become a necessity for NatureMapr.

I know that there have been reports from users noticing some sightings not showing up where they should, or at least, not quickly enough. I want to assure you that this is only a display issue and that the underlying "source of truth" data is fully intact.

While Rohan's important work of redesigning the maintenance service takes place, we expect that you may encounter a few such issues. I also expect this may continue for a couple of weeks. Your data isn't actually missing, it simply may not have been "published" by the maintenance service yet, but it will be soon.

Thanks so much for your patience and continued support. It means a tremendous amount to both Rohan and I.

Please know that we are fully onto it - the result will be a hugely scalable citizen science platform that hums along and is able to support the next wave of NatureMapr's continued growth.

7 comments

   9 Apr 2019
Does the system weed out duplicates, created when people upload a sighting for the second time? Great work guys! Mike L
   10 Apr 2019
You're both doing a great job. With such massive data there's sure to be a few glitches at times.
   10 Apr 2019
Duplicates are prevented Mike at the point of upload, i.e. they are not accepted. That is, same species at the same lat/long location on the same exact date/time is considered a duplicate record.
   10 Apr 2019
Alison I am extremely lucky to have Rohan join the team, as we are well beyond a 1 man operation now.
Illilanga wrote:
   11 Apr 2019
Re duplicates. I noticed the duplicate issue also applies if you use the same photo (or the same last photo in my case) for two sightings (i.e. same meta data). I got around it by uploading my photos in a different order. Took me a few times before I twigged what was happening. Great work you two! Aaron, you can blame us "super users" for all your pain. ;)
   11 Apr 2019
Keep up the great work, lads.
RodDeb wrote:
   11 Apr 2019
Thanks for all of the hard work you all are doing, all of you, Aaron, Rohan and all of the moderators you all do an amazing job. Thank you very much we really appreciate all of your hard ongoing work.

Please Login or Register to comment.

Author information

AaronClausen
AaronClausen

Founder, NatureMapr
Canberra Nature Map Atlas of Life in the Coastal Wilderness Frogwatch ACT and Region Atlas of Life Budawang Coast Southern Highlands Nature Map Albury Wodonga Nature Map

Powered by NatureMapr |  Noosa Shire Nature Map operates under Creative Commons Attribution 3.0 Australia |  Privacy