Write a backend microservice in GAE to handle ingestion from Meetup.com

[Splaktar]

This should use the service: backend annotation in GAE so that it can run in parallel with our frontend web service. It may make sense to make this a separate repo (hub-ingest) or something like that. It should use the same production MongoDB and GDG-X Cloud project, but in no other way should it be coupled to the existing Hub backend.

Phases

1. The first piece of this work is to get this running locally and ingesting a single chapter's Meetup.com events into your local MongoDB
2. The second phase is to ingest and update all GDG chapters with data from Meetup.com into your local MongoDB
3. The third phase is to ingest events for all chapters into your local MongoDB. We likely want to look at using PubSub and GAE Cron for this, but we should avoid our current pattern of making as many API calls as fast as possible.
4. The fourth phase is to ingest other metadata from Meetup.com like organizers, etc. into your local MongoDB
5. The final phase will be the last round of code reviews, testing, and then production deployment.

[VikramTiwari]

Why not a Cloud Dataflow job to do this? Is the aim to stay as close to GCP or be platform independent?

[Splaktar]

Cloud Dataflow would be fine, does it fit well with the pattern of ingesting via Meetup.com REST APIs? I personally haven't used it yet. Would the code have to be in Java/Scala?

The project started fairly GCP independant as it needed to be hosted freely on Heroku and OpenShift. But now that Google DevRel is covering GCP costs, it is no problem to use GCP-specific tech and designs.

[VikramTiwari]

Yeah! Though Dataflow is not part of the free plan. I plan on writing using Python and using Apache Beam SDK, so there shouldn't be any issue with dependence. Though we might have to run it as CRON script until either GCP can sponsor gdg-x or Dataflow comes as part of free platform.

Interesting would be to see Cloud Functions support python and then we should be able to run it there too, which is under free section.

[Splaktar]

@VikramTiwari It doesn't need to be free. It should of course be cost effective, but we aren't restricted to free as Google DevRel is covering our expenses.

The current ingestion (and entire project) is written in JavaScript (MEAN stack), so bringing in another language (Python) is not ideal. But if it is much easier to do in Python and only the Meetup.com API ingestion piece needs to be in Python, then that should be OK.

Cloud Functions are NodeJS only at this point. It would be possible to run it as a GCF using GAE Cron to call a service that sends a PubSub message. Then the GCF would listen to that topic and do ingestion when the message comes in. I don't really think that this is an ideal solution though.

What do Dataflow jobs look like? Do you deploy them to Dataflow or are they just GAE services that call Dataflow APIs? Can you detail some of what the Dataflow design would look like please?

[VikramTiwari]

I looked into Dataflow and it seems like python support is very limited as of now. If the complete app's backend is NodeJS then a cloud function with GAE Cron seem like a good design. Why do you think this as non-ideal? We can even make the function parameterized and HTTP execution based so that update of database happens can also happen in the realtime. Let me know your thoughts.

[Splaktar]

@VikramTiwari We may want to understand the Meetup.com API quotas better before settling on a design.

• Should we process all chapters, events, organizers, etc as a single ingest job (and single point of failure)?
• Should we process them in separate jobs (more complexity)?
• Should we process them all at once, in batches, or individually (separate PubSub message for each)?

It seems like we should ingest once per chapter (approx every 6 hours if possible), with that ingest updating the chapter's details, events, and organizers. We could chain these into separate steps via PubSub and GCF.

Current stats:

• Global Meetups tagged with GDG - 299 chapters
• Global Meetups registered with the GDG Pro account - 146 chapters
• Stats from DevSite - 590 chapters, 3,600 events in last 6 months

The second is the official number and it is expected to grow beyond 500 on May 1st. Will Meetup.com be OK with us hitting them with 1500-2500 API requests every 6 hours?

As for deciding on GCF vs GAE, what are the pros and cons?

GAE standard backend service

Pros

• Autoscaling
• Simple integration with GAE cron
• Works well with PubSub
• Works with Cloud Endpoints to enforce security/quota
• Scale to 0

Cons

• Must use Java/Python
• Must use different Google Cloud APIs

GAE Flex backend service

Pros

• Autoscaling
• Simple integration with GAE cron
• Works well with PubSub
• Works with Cloud Endpoints to enforce security/quota
• Same NodeJS language we use now
• Works with google-cloud-node libraries

Cons

• No scale to 0

Google Cloud Functions (GCFs)

Pros

• Autoscaling
• Scale to 0
• Works well with PubSub
• Same NodeJS language we use now
• Works with google-cloud-node libraries

Cons

• No way to call directly from GAE cron
• Does not support Cloud Endpoints to enforce security/quota

What else am I missing here?

[pmoosh]

For now - watching.

[BrockMcKean]

The hourly limits are included in the response as X-RateLimit headers described by https://www.meetup.com/meetup_api/#limits
However, https://groups.google.com/forum/#!topic/meetup-api/7xpoMZF43CQ seems to suggest that the default is 200/hr (+3 years ago), which means the upper limit every 6 hours would be 1200, 300 below your projected minimum of 1500. Assuming this is correct, it will require 3 IP addresses based on @Splaktar 's consumption projections.

I don't understand Cloud Endpoints enforced quotas well enough to give a definitive analysis, but given that the limits are in the header, it would seem best not to run this on a static cron, but on a callback based on the X-RateLimit header. I'll take a look at integrating https://www.npmjs.com/package/meetup-api tomorrow.

[Splaktar]

@BrockMcKean great, thanks for looking into the Meetup.com API, those details are quite helpful.

As far as integration, the thought so far has been that this Meetup.com ingestion could take place in a separate GAE service. But I didn't really get a response to #101 (comment) in order to decide how we should finalize the design.

I don't think that using Cloud Endpoints is really required for doing ingestion from the Meetup.com API.

Should we design the system to publish tasks to PubSub for each update that is needed, then pull from PubSub (and process the ingest) at rate that is acceptable to Meetup.com (i.e. 200/hr)? I'm not sure that Cloud Functions can be configured in this way (I think that they spin up every time a message is published on a topic).

[Splaktar]

@BrockMcKean Oh and if you are trying to test adding the Meetup.com ingest to the existing project, please use this branch since it contains some pretty significant changes over develop.

We could certainly keep the Meetup.com ingest as part of this same project which will be deployed to GAE Flex, then just setup a cron job to call a task that publishes to PubSub all of the chapters/events that haven't been updated in x time (around 6-12 hours is what we've done in the past).

[VikramTiwari]

As for this comment, I think GCF should be our choice since I am unable to understand the need for Cloud Endpoints security there. If the requests to Meetup API are being made from our side and result comes to us and gets stored in our database, where does the part of Cloud Endpoints even come in picture?

Regarding function invocation, a simple pub-sub request should be more than enough to trigger it, which should be really easy to integrate with a CRON based task on Hub itself.

[Splaktar]

I agree that Cloud Endpoints aren't needed for kicking off the Meetup.com ingestion.

@VikramTiwari Are you envisioning only a single event being sent to PubSub to 'update all data from Meetup.com API'? How do you envision things working after the GCF is triggered, especially in regards to a 200 request per hour Meetup.com API quota?

[VikramTiwari]

I am not sure if 200 requests per hour limit is still applicable. Most of the API teams have started using a rolling average to limit unethical usage of their APIs which works on various constructs such as time, IP address, credentials etc. If there is an upper limit then we will need to tune our code. The way I envision code to be is to run these ingestions based upon data classification. A very basic classification is:

• Chapter data, ~12h
• Event data, ~6h
• Event stats, ~3h
• Anything else, ~12h

This is just to demonstrate, I am not sure what is best breakup or what is the best time duration. But we should not use a blanket value for all the data.

Also as @BrockMcKean mentioned, X-RateLimit header will be pretty useful. If we are approaching limits, we can slow down our queries. Though this might not be true in case of async.parallel queries.

Also, I am pretty sure that if we ask meetup support, they can provide special higher level quota for our application, but first let's make sure that we are optimized.