{"id":1220,"date":"2010-05-01T16:18:53","date_gmt":"2010-05-01T06:18:53","guid":{"rendered":"http:\/\/www.somethinkodd.com\/oddthinking\/?p=1220"},"modified":"2010-05-01T16:20:10","modified_gmt":"2010-05-01T06:20:10","slug":"how-pythons-urllib-bit-me","status":"publish","type":"post","link":"https:\/\/www.somethinkodd.com\/oddthinking\/2010\/05\/01\/how-pythons-urllib-bit-me\/","title":{"rendered":"How Python’s urllib2 Bit Me"},"content":{"rendered":"

Continuing – not for too long, I hope – in the series of the dangers of making assumptions about Python libraries while programming.<\/p>\n

Suppose, hypothetically, you want to fetch a feed from a web-server using some Python code.<\/p>\n

You would probably go and use urllib2<\/code> which makes it straight-forward, if you want it to be, or sophisticated if you need it to be. <\/p>\n

Now, network programming is notoriously unreliable, so you should give some quick thought to that. You should have some exception handlers in place to handle timeouts and DNS problems and the like. Note that urllib2.URLError<\/code> includes some “non-error” exceptions, like authentication requests, but you’re not expecting those, so they count as errors here.<\/p>\n

On the subject of timeouts, you can now (since Python 2.6) override the default with a parameter.<\/p>\n

The optional timeout<\/code> parameter specifies a timeout in seconds for blocking operations like the connection attempt (if not specified, the global default timeout setting will be used). This actually only works for HTTP, HTTPS, FTP and FTPS connections. [Ref: urllib2 manual, Python 2.6.4]<\/p><\/blockquote>\n

So you care how long it is? No, you are happy to use whatever the typical default is for fetching a web-page – what would that be? 2 seconds? 10 seconds? 20 seconds? As long as it isn’t silly, like 5 minutes, you’ll be fine, so, hypothetically, leave that parameter empty.<\/p>\n

Whoops. You shouldn’t have done that! Not even hypothetically.<\/em><\/p>\n

The urllib2<\/code> package doesn’t specify the default timeout in this situation. It sits on the httplib<\/code> library, and it punts that issue down. It assumes that users of the URL library have the same needs as the users of the HTTP library.<\/p>\n

But httplib<\/code> doesn’t specify the default timeout either. It sits on the socket<\/code> library, and it punts that issue down. It assumes that users of the HTTP library have the same needs as the users of the Socket library.<\/p>\n

The socket<\/code> library is a fairly generic library used by many users. It doesn’t know whether it is being used to fetch web-pages across the web, or for interprocess communication by parts of the same application. So, it sets the default timeout as… forever<\/em>.<\/p>\n

Somehow, that default doesn’t seem quite so reasonable for the library responsible for fetching web-pages.<\/p>\n

It is a nasty little problem, as an unresponsive remote web-server will freeze your application’s thread… but your application thread won’t realise it is having problems, so it will never log an error to ensure you get an email. Nor will it crash, so your watchdog process can restart it. No, it just quietly hang your server application at, say, 29 minutes past midnight on a Saturday morning. Purely, a hypothetical example, of course.<\/p>\n

Now, this is a low-level API-usage error that will hopefully be noticed in code-review. Err.. Whoops, no, it wasn’t… err… hypothetically, I mean.<\/p>\n

That’s a problem, because it isn’t a problem you will likely find in unit-testing. You may be diligent and mock out the urllib2<\/code> to make sure your code handles timeout errors correctly, but are you going to set-up a mock web-server to timeout and see whether timeout errors are actually raised?<\/p>\n

Nor is it likely to come up during system-testing. How often do web-servers fail that way?<\/p>\n

Actually, that is a question I can answer. If you were to have such a bug, it wouldn’t appear until the 2,879,702th call to the relevant function in urllib2<\/code>, so when it did happen, you would be quite surprised that a piece of code that you thought was very stable could bring down your production server for many hours until it was noticed.<\/p>\n

Hypothetically, that is.<\/p>\n","protected":false},"excerpt":{"rendered":"

In which Julian proposes a hypothetical – how could the default parameters on urllib2<\/code> bring down an application 29 minutes past midnight on a Saturday morning?<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_s2mail":"","footnotes":""},"categories":[23,34],"tags":[],"class_list":["post-1220","post","type-post","status-publish","format-standard","hentry","category-based-on-a-true-story","category-software-development"],"_links":{"self":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/posts\/1220","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/comments?post=1220"}],"version-history":[{"count":6,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/posts\/1220\/revisions"}],"predecessor-version":[{"id":1226,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/posts\/1220\/revisions\/1226"}],"wp:attachment":[{"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/media?parent=1220"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/categories?post=1220"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.somethinkodd.com\/oddthinking\/wp-json\/wp\/v2\/tags?post=1220"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}