PickMySixBot
If you have seen PickMySixBot/1.0 in your logs, this page is for you. It
says exactly what it does, and how to stop it.
What it is
A small set of Node scripts that read published lottery results and store them as an open archive. We hold 8,508 draws across 15 games in 7 countries, published under CC BY 4.0 on our data page. There is no crawler in the general sense: it fetches a specific list of results pages and exports, and nothing else.
How it identifies itself
PickMySixBot/1.0 (+https://pickmysix.com/bot)Some operators serve a different response to a non-browser agent on paths that carry no restriction. Where that happens we present a browser user agent instead, and we say so here rather than only in a code comment. What does not change: the rate limit, the crawl-policy check below, and that we never sign in.
How we handle robots.txt
Every request is checked against the host's robots.txt before it is made,
and the policy is read once per host per run. This used to be a claim in a code comment
with nothing behind it — each path had been checked by hand when its collector was
written, and nothing would have noticed a host changing its mind. It is now enforced.
| Rules published | We follow them. Longest match wins, and a group naming us takes precedence over the wildcard. |
|---|---|
| 404 or 410 | No policy exists, so nothing is restricted. This is what the standard says. |
| 401 | We fetch nothing. A policy that exists and is withheld is not the same as one that does not exist. |
| Server error, or no answer | We fetch nothing this run and try again tomorrow. |
| 403 | See below. |
A 403 needs explaining, because it is the awkward case. Two of the hosts
we read sit behind a content-delivery network that denies /robots.txt to
everyone, a browser included. No policy is being published there at all, so there is
nothing to honour — but "no policy" and "a policy we are not allowed to see" look
identical from the outside.
So a 403 stops us by default. It only proceeds for a host somebody has looked at and written up, in a short list in the code that records what was checked and on what date. A host we have not considered is refused, and stays refused until a person decides. If that is your host and you would rather we did not, the paragraph below is all you need.
What it will never do
- Create an account, or accept a terms dialog. Clicking through a consent screen turns a public page into a contract, and we will not do that on anyone's behalf.
- Route around a block. If a host refuses the connection, that is an answer. We do not change IP address, use a proxy or a relay, and we do not retry a refusal on a schedule. Four countries are missing from this site for exactly that reason, and they are staying missing.
- Take a compiled archive. We read results from the operator that ran the draw or from a government open-data service. Somebody else's collection is their work, not a fact in the public domain.
- Log in, submit a form, or touch anything behind authentication.
How often, and how hard
| Rate | At most one request per second to any one host, always. |
|---|---|
| Schedule | Twice a day. Most runs fetch one file. |
| Concurrency | None per host. Requests are serialised. |
| Backoff | A 429 or a 5xx backs off and retries a few times, then gives up for the day. |
| Response cap | 40 MB. Anything larger is refused rather than downloaded. |
We archive the exact bytes we received before parsing them, so that if a page changes and our parser starts producing nonsense we can re-read the history rather than come back and fetch it from you again.
How to block it
Add this to your robots.txt and we will stop, on the next run:
User-agent: PickMySixBot
Disallow: /You do not have to ask us first and you do not have to explain. If you would rather email, [email protected] reaches a person and we will remove the source. We would rather cover fewer games honestly than argue about it.
Who we read
Results come from these operators and open-data services: Allwyn, Caixa Econômica Federal, Interprovincial Lottery Corporation, MUSL, Mega Millions Consortium, Mizuho Bank, Multi-national (drawn in Paris), Premier Lotteries Ireland, SELAE, and the New York State Gaming Commission's open data programme. Each game's page names its own source and states how far back our record goes.
If something here is wrong
If our archive holds a result that does not match what you published, tell us the game and the date and we will fix it and say what changed. Contact.