Blog: MT-RefSearch v0.7

December 12th, 2002 by Richy B. Leave a reply »

[MT-RefSearch]Woo! After several days of work, I’m proud to announce the release of MT-RefSearch v0.7. Inspired and based on the original version by Eliot Landrum, this new version not only recognises Google but also over 240 other search engines. (The “official” homepage of MT-RefSearch is still on Eliot’s site by the way). You can also see Eliot’s announcement of the release.

Basically, MT-RefSearch is a PHP script and Javascript which you can include MovableType powered site, and when people come to your site from one the 243 listed search engines, they will be presented with the most appropriate search results from your site.

Any comments, criticisms, problems, ideas etc – please feel free to comment here and I’ll do my best to assist. I will be adding even more search engines to the ‘scan list’ (I did mention to Eliot I was aiming to get it over the 500 mark, but ‘upgrades’ to the main bulk of the code stopped me getting that far) – so keep checking either my blog or Eliot’s “official” MT-RefSearch site for future updates….

MT-RefSearch works using server-side code (PHP) to search the database and client-side code (Javascript) to allow people to highlight the text if the search results appear on the page they are viewing.

Requirements

Installation

There are two files you need to install to ensure MT-Refsearch works correctly:

Once you have installed those files, you will need to ensure that MT-RefSearch is ‘called’ from each file on your site. To do so, add a line similar to:

<php include_once “/home/blog/mtrefsearch.php”; ?>

into your “Main Index” and “Individual Entry Archive” templates.
Rebuild your site and away you go!

Installation of Lookup code

[return to installation]
To ensure the MT-Refsearch can make the correct links to your archive file, it needs to have a ‘lookup’ file to cross-reference all the entries against. This file is rebuilt automatically by MovableType every-time you publish a new entry so you don’t have to worry about keeping it up to date yourself.

  1. Login to your Movable Type administration page and go to “Templates”
  2. Go to “Create new index template”
  3. Ensure the ‘Output file’ is called “mtrefsearch_lookup.html” and the “Rebuild this template automatically when rebuilding index templates” box IS ticked. It is advisable to give the file a ‘human recognisable’ template name of something like “MT-RefSearch: Lookup”
  4. Copy the contents of mtrefsearch_lookup into the “Template Body”
  5. Save the file
Installation of MT-RefSearch

[return to installation]

  • Download and open in a text editor (such as Notepad or Editplus) the mtrefsearch file.
  • Change the following lines near the start of the file:
    • Line 21: $blogid = "1";
      This needs to reflect the ID number of your blog. The easiest way to find this out is to login to MovableType and click on ‘Manage Blog’ – look at the URL bar and you will see something like: mt.cgi?__mode=menu&blog_id=1 – the number at the end is your “blog id”.
    • Line 23: $mtcfghome="/home/blog/";
      This needs to point towards the folder/directory where ‘mt.cfg’ resides (the file you needed to configure when you originally installed MovableType).
  • Login to your Movable Type administration page and go to “Templates”
  • Go to “Create new index template”
  • Set the ‘Output file’ to be a name you can remember ending in ‘.php’: for this tutorial the ‘Output file’ will be called ‘mtrefsearch.php’. (Note: this file name MUST be the same name that you put in the include_once line above.)
    Ensure the “Rebuild this template automatically when rebuilding index templates” box is NOT ticked. It is advisable to give the file a ‘human recognisable’ template name of something like “MT-RefSearch: Mainfile”
  • Copy and paste the amended mtrefsearch.txt file into the “Template Body”
  • Save the file
  • (Haven’t I been a busy boy though – not only did I give the script a total overhaul, I also wrote these instructions and constructed the nifyish little graphical logo: it’s meant to be a combination of the Google and Movable Type logos with the name of the script added with a bit of perspective. Ok, perhaps not the most original or best looking of logos – but I ain’t a graphic designer! 🙂 “Sexy programmer/scripter wonder-boy” – yep, “graphic designer” – nope.

    This post is over 6 months old.

    This means that, despite my best intentions, it may no longer be accurate.

    This blog holds over 12 years of archived content - during that time, I may have changed my opinion of something, technology will have advanced (and old "best standards" may no longer be the case), my technology "know how" has improved etc etc - it would probably take me a considerable amount of time to update all the archival entries: and defeat the point of keeping them anyway.

    Please take these posts for what they are: a brief look into my past, my history, my journey and "caveat emptor".

    25 comments

    1. Kristen says:

      well, if i ever manage to get my MT to run on mySQL, then i’ll give this a go…

      very clever, by the way!

    2. Jon says:

      Great tool for MT!

      One likkle thing tho’ … you’ve missed a certain question mark in the include code! infront of the php bit =)

      Something that may catch a few people out

    3. Jon says:

      DOH! Stupid me!

    4. DJ says:

      Hi Richie,

      Really like this idea and tried to implement on my site – despite being a technical ignoramus!

      1 question – if my index page is an shtml, can I still use the PHP code? I’m thinking that’s where I’ve gone wrong, it probably all needs to be php-based, right?

      Thanks,
      DJ

    5. Sammael says:

      I also get those errors:

      Warning: Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2) in /web/script/sammael/themaelstrom.net/mtrefsearch.php on line 693

      Warning: MySQL Connection Failed: Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2) in /web/script/sammael/themaelstrom.net/mtrefsearch.php on line 693

      Any hints on how to fix that ? Thanks !

    6. Tyler Smith says:

      I made a slight modification to the “AddSearchTerm” function that makes the searching slightly more functional by adding the “entry_keywords” as a field to search as well. This way, if you are getting a lot of references from search engines to posts with keywords that are not showing up in the search listing (but nevertheless exist), one can add whatever keywords are missing to the “keywords” field in MT and resolve the problem. As I said, a slight mod, but hey, can’t hurt right? 😛

      function AddSearchTerm($sql,$word) {
      // help build our search query
      $sql.=” ((entry_text LIKE ‘%”.mysql_escape_string($word).”%’) OR “; // search body
      $sql.=” (entry_text_more LIKE ‘%”.mysql_escape_string($word).”%’) OR “; // search extended
      $sql.=” (entry_title LIKE ‘%”.mysql_escape_string($word).”%’) OR “; // search title
      $sql.=” (entry_keywords LIKE ‘%”.mysql_escape_string($word).”%’)) “; // search keywords
      return $sql;
      }

    7. Gert says:

      First of all: wow! Great tool!

      I managed to implement it on my blog. It works fine on the mainpage. But it doesn’t list the findings on archive files. What am I doing wrong?

      To see what I mean with the listing of findings on archive files, click on the link starting with “Totally Flabbergasted” on http://www.google.nl/search?q=winny+de+jong+panorama&ie=UTF-8&oe=UTF-8&hl=nl&btnG=Google+zoeken&lr=lang_nl

      When someone finds me on a searchengine and hits the mainpage: click on the link starting with “Totally Flabbergasted” on http://www.google.com/search?sourceid=navclient&hl=nl&q=christiaan+albers

      My archivefiles are stored in /archief/

      My blog is in Dutch, but I don’t think that’ll be a problem.

    8. Excellent script! Kudos to you and Eliot both. Another use people may find handy (one I immediately hacked in): For sites that run a mix of MT and static content (for example, I have static pages for descriptions of engineering projects, as the HTML is more intensive and the content doesn’t change), eliminating the REFERER check and just setting $keywords yourself makes for a nice way to integrate in any MT posts along similar lines — in my case, the “project in progress” items.

    9. Black_Knight says:

      This is a really nice Mod for MT, but it brings up a natural yet vital question: Is it actually appropriate and needed?

      The purpose of a search engine is to index content from any document and enable users to find information matching their query from their index of millions (billions in the case of Google and FAST) of documents crawled each month.

      In short, the engine should have already done the relevancy part and be sending visitors to the exact page/document that matches their needs. Mostly this will be hits straight through to your archived pages rather than your homepage.

      Using this mod, you run the risk of falling foul of a common search engine rule – you must not show different content to a spider than you would to the surfer arriving from that search engine. See Google’s Webmaster Guidelines.

      Quote: Don’t deceive your users, or present different content to search engines than you display to users.

      Quote: Don’t employ cloaking or sneaky redirects.

      By presenting one set of content to the search engine spider when it originally indexes the document, and different content to the user who later arrives as a result of what the spider found there matching their query, you are entering the realm of cloaking, and quite possibly accidentally getting in the way of a perfectly good referral for no good reason.

      I’m saying this not as an indictment of the mod. The mod is cool. What I’m saying is be very careful in how and why you use it, because a ban from that search engine would negate the whole point of having the mod in the first place. And bans for cloaking happen every day.

    10. Black_Knight: It doesn’t present different content TO the spiders, just slightly to the users from those search engines. It’s a bit like the “Google Highlight” feature and the search results you get if you click from Google to webmasterworld or similar.

      It’s not cloaking, but just helping users to navigate. If the user launches a new window from the search results, blocks the browser referer, is behind a very strict firewall or types in the URL – then MT-RefSearch will just site quietly. However, if you hit it directly from a search result, then it’ll try and help you find the content you were looking for on the page. I’m sure if Google et al did class it (and similar scripts) as even being slightly “cloakish”, the news would have reached me – but as it is, I keep on appearing high on Google for all manner of search words and I’ve got a nice PR value of 6 without any “real” search engine optimisation etc.

    11. pat says:

      once implemented and a link is followed from google, if i view my code, i am getting the following error:

      any idea why this is occuring? the php includes and paths are pointing towards the proper directories. strange…

      thanks

    12. Pat:

      This actually had me puzzling for quite a while: I first thought “bad/odd MT installation”, then I thought “odd web server” configuration – but neither seemed quite right: so I went nosying…

      It seems MT-Refsearch isn’t quite as forgiving on the mt.cfg file as it should be – looking at your configuration file shows that you’ve got additional spaces at the start of the Database, DBUser and DBHost lines: if you delete the extra spaces at the start of the lines then the script should start working properly (if the script was unable to read the mt.cfg file at all, it would have thrown the “Unable to open …mt.cfg error”).

    13. pat says:

      super! i now have it installed and close to working. the only trick is that the generated list of links all point to my main blog. they do not produce links to my individual archived posts. i had to edit the mtrefsearch_lookup.html file slightly to incorporate my correct archive paths. the generated file has the proper links in it, but the mt-refsearch.php generated output on my site does not show the proper archive paths. any ideas?

    14. After installing this search, I was originally getting the same error as Sammael:

      Warning: Can’t connect to local MySQL server through socket ‘/var/lib/mysql/mysql.sock’ (2) in //mtrefsearch.php on line 693

      The problem is that the database server name is hard coded as “local” in the PHP file.

      If your mysql server does not run at localhost, then your will have to change “local” in line 690 to your mySQL server name.

      Now it works fine for me.

    15. kredite says:

      First of all: wow! Great tool!

      I managed to implement it on my blog. It works fine on the mainpage. But it doesn’t list the findings on archive files. What am I doing wrong?

      To see what I mean with the listing of findings on archive files, click on the link starting with “Totally Flabbergasted” on http://www.google.nl/search?q=winny+de+jong+panorama&ie=UTF-8&oe=UTF-8&hl=nl&btnG=Google+zoeken&lr=lang_nl

      When someone finds me on a searchengine and hits the mainpage: click on the link starting with “Totally Flabbergasted” on http://www.google.com/search?sourceid=navclient&hl=nl&q=christiaan+albers

      My archivefiles are stored in /archief/

      My blog is in Dutch, but I don’t think that’ll be a problem.

    16. Rick Klau says:

      Hi – I’ve noticed that MT-Refsearch’s links to my individual pages are creating links not to the permalink for the post, but to the directory the post lives in. Any idea why this would be? It was working fine, the only thing I could think of is that I’ve upgraded to MT 2.66. Is that it? Anything else to check?

    gamy-dance
    %d bloggers like this: