DonorDrive AMA w/ the Extra Life Community - Page 2 - Extra Life JSON Code Discussion & Sharing - Extra Life Community Hub Jump to content
herobyclicking

DonorDrive AMA w/ the Extra Life Community

Recommended Posts

Greetings, @Alex Lewis!

 

That is indeed the case right now, since the API is exclusively public and we consider that a "private" field to the fundraiser.

 

We're working on adding support for authenticated requests to the API, but there's nothing firm with regards to a timetable.

 

Sorry for the inconvenience (for now),

Tim

Share this post


Link to post
Share on other sites

Hey there,

 

Since the DDOS, my Python program is getting a 403 when trying to access the API. Is there something new that needs to be done?

Share this post


Link to post
Share on other sites

Sure. I did a quick sample call:

Python 3.7.4 (default, Jul  9 2019, 16:32:37)
[GCC 9.1.1 20190503 (Red Hat 9.1.1-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib.request
>>> url="http://www.extra-life.org/api/participants/348774"
>>> urllib.request.urlopen(url)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib64/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

 

And if I visit that url in my browser it works. Now, the browser changes it to HTTPS, so I wanted to double-check that:

 

>>> url="https://www.extra-life.org/api/participants/348774"
>>> urllib.request.urlopen(url)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib64/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

 

And just to make sure it's not my machine:

 

>> url="https://www.github.com"
>>> urllib.request.urlopen(url)
<http.client.HTTPResponse object at 0x7f4ba7254f10>

 

Thanks!

Share this post


Link to post
Share on other sites

Interesting. Can you confirm that you're hitting the API via https? You *must* use https to make calls to the Public API.

 

If that's not it, I might need some actual code to look at. Do you have a repo for this anywhere?

Share this post


Link to post
Share on other sites

Hey Tim,

 

Is this a new requirement post-DDOS? I've been using the code without HTTPS all 2019 (and previously, of course). My code https://github.com/djotaku/ELDonationTracker does not use HTTPS, but that's why I tried it on the commandline in my previous post

 

Quote

>>> url="https://www.extra-life.org/api/participants/348774"
>>> urllib.request.urlopen(url)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib64/python3.7/urllib/request.py", line 222, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib64/python3.7/urllib/request.py", line 531, in open
    response = meth(req, response)
  File "/usr/lib64/python3.7/urllib/request.py", line 641, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib64/python3.7/urllib/request.py", line 569, in error
    return self._call_chain(*args)
  File "/usr/lib64/python3.7/urllib/request.py", line 503, in _call_chain
    result = func(*args)
  File "/usr/lib64/python3.7/urllib/request.py", line 649, in http_error_default
    raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 403: Forbidden

 

and that failed for some reason. But github with https didn't give me that warning.

 

edit to add, not sure what this guy's code is doing, but it's another complaint of python not workign:

 

Share this post


Link to post
Share on other sites

did a bit of googling and ... maybe I need to add a header. Will research and get back to you.

OK, yup. That fixes it.

Share this post


Link to post
Share on other sites

Two things:

1) Now that I have headers, it works even with http (doesn't need https)

2) the amount of places I had to change it drives home how much I need to refactor my code!

Share this post


Link to post
Share on other sites

Different issue I'm noticing. Maybe I'm hitting the API too much post DDOS? But I amd getting connection timeout errors. If you could take a quick peek and let me know what you think -

 

https://github.com/djotaku/ELDonationTracker

 

I haven't changed anything in extralifedonations.py since before the DDOS.  Lines 172-203 govern the loop. I have it sleeping 30 seconds between each iteration. However, within that 30 seconds I hit at least 4 times - participant, donors, team, team donors. Is that too much now? Is there a more efficient way that's more compliant in the post-DDOS world?

Share this post


Link to post
Share on other sites

You *shouldn't* be, but I can't rule it out. If you'd like to DM me with some network info, I can take a look. You can find me over on the #donordrive-support Discord channel.

 

As far as polling strategy, let me walk you through what we do internally. We leverage the `etag` + `if-none-match headers` to determine the need to poll "subordinate" endpoints. Using the participant + participant donors model above, you can poll the /participants/XXX endpoint on your 30 second intervals furnishing the `if-none-match` header. If you get a 200, then that means something changed (you'll get a 304 if nothing has changed), you can subsequently compare `participant.sumDonations` with your previous iterations `participant.sumDonations`. If the values aren't equal, you know there are new donations to fetch, otherwise, you dont even need to bother polling /donations.

 

The ancillary benefit of this route, is that you don't have to assume a refresh (and subsequently the parsing/rendering overhead) of your assets on every polling interval.

 

Hope this helps,

Tim

Edited by Tim [DonorDrive]
updated to correct status code

Share this post


Link to post
Share on other sites

I created a discord account. And while it seems to be Slack-like or IRC-like - I've never used it before. I take it I need to find the server you're on the find your channel. What server are you on?

Share this post


Link to post
Share on other sites

Oh I'm sorry, I just assumed you were on there... Hop onto the ExtraLife4Kids server. There's a #donordrive-support channel in there. I'm "@Tim [DonorDrive]"

Share this post


Link to post
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


×
×
  • Create New...