How to Add Search Engines that use POST to Search.ini

note: Please see SearchINIEditing for a general tutorial on using search.ini, or OpSED if you want more information on the search.ini editor.

Search engines can use two methods to send your query to their site to start the search. The first is called a GET, and it uses the 'address bar' to send information. An example of this is Google, run a search then look at your address bar, you will see a string of values seperated by ampersands (&). Google uses this information to perform the search. OpSED contains a very useful URL wizard which allows users to easily add search engines that use GET without having to know very much about how it works.

Other search engines use a more secure, but more difficult to decipher method called POST. This system hides the information in the header that gets sent to the server. You never see this information, and so it is difficult to work out how to configure search.ini to work. It is possible, but you will need an external tool to uncover this information. TheProxomitron is ideal for this, as it allows you to easily snoop on the hidden browser/server conversation. To do that, simply select "Log Connections" from the interface, and then in the log menu, turn ON "View Posted Data". This allows you to see a list of the posted data sent. Another excellent tool to allow snooping of this hidden conversation is HttpSpy, which specialises in logging this.

I will use PubMed as the example here. Both TheProxomitron and HttpSpy are used as a proxy, and you simply get Opera to use them as such. I will show the results using TheProxomitron, but the process is very similar for HttpSpy. First, go to PubMed, then open up TheProxomitron's "Log Conections" and turn on "View Posted Data". Now switch to Opera and enter the search expression "perception timing". Switch back to the "Log Connections" window and at the top you will see something like:

POST /entrez/query.fcgi?SUBMIT=y HTTP/1.1
User-Agent: Opera/7.23 (Windows NT 5.0; U)  [en]
Host: www.ncbi.nlm.nih.gov
Accept: text/html, application/xml;q=0.9, application/xhtml+xml;q=0.9, image/png, image/jpeg, image/gif, image/x-xbitmap, */*;q=0.1
Accept-Language: en
Accept-Charset: windows-1252, utf-8, utf-16, iso-8859-1;q=0.6, *;q=0.1
Accept-Encoding: deflate, gzip, x-gzip, identity, *;q=0
Referer: http://www.ncbi.nlm.nih.gov:80/pubmed/
Cookie: WebEnv=0N0V7ZS7H9DZbIXIM9eP2JAZbr7uknl6OXQ2u9NMMOJRoUdVw2A0
Cookie2: $Version=1
TE: deflate, gzip, chunked, identity, trailers
Content-type: application/x-www-form-urlencoded
Content-length: 139
Connection: keep-alive
Posting 139 bytes...
WebEnv=0N0V7ZS7H9DZbIXIM9eP2JAZbr7uknl6OXQ2u9NMMOJRoUdVw2A0
db=PubMed
orig_db=PubMed
term=perception+timing
cmd=Search
cmd_current=
x=0
y=0

The bit that is important is that bit AFTER the line Posting 139 bytes.... The first long string is probably specific to this session (normally signified by such random looking strings of numbers and text), but the rest of the lines are useful to us. Search.ini expects everything on one line, and so you must put & between each line and remove the line breaks. This gives us the following query:

db=PubMed&orig_db=PubMed&term=perception+timing&cmd=Search&cmd_current=&x=0&y=0

Now replace the term=perception+timing to term=%s (search.ini uses %s for the entered term) and you have your worked out your POST query for search.ini / OpSED. HttpSpy is even easier to use because it give the POST in the correct format:

db=PubMed&orig_db=PubMed&term=perception+timing&cmd=Search&cmd_current=&x=0&y=0

All you have to do here is add %s and you're set. So, now you can decipher search engines that use either GET or POST when customising your search.ini in Opera.

Alternative Method

If you don't want to install a separate program to find out POST queries. Here is an alternative method that might work.

The bookmarklet below will run through a page and convert all forms from POST to GET. Once this is done you can submit the form as usual and copy the query data as you would for any GET query.

Note that many forms actually don't care how they receive their query data so it's entirely possible to leave the query as a get. But for compatibility's sake if a form started life as a POST you should keep it as one when adding it via OpSED

Form POST to GET (author: Jesse Ruderman)

Below is an untested version of the bookmarklet that should work with a page that uses frames. It was created using a standard template for making frames compatible bookmarklets but this is no guarantee that it will work in all situations (if someone knows a good place to test it feel free and mention here whether it worked).

Form POST to GET

Categories:

CategoryOpera CategoryTutorial

Back Links:
There are 34 comments on this page. [Display and/or add comments]