Continuing to chapter 4 of Programming Collection Intelligence (PCI) which is implementing a search engine.
I may have bitten off a little more than I should of in 1 exercise. Instead of using the normal relational database construct as used in the book, I figured, I always wanted to have a look at Neo4J so now was the time. Just to say, this isn't necessarily the ideal use case for a graph db, but how hard could to be to kill 3 birds with 1 stone.
Working through the tutorials trying to reset my SQL Server, Oracle mindset took a little longer than expected, but thankfully there are some great resources around Neo4j.
Just a couple:
neo4j - learn
Graph theory for busy developers
Since I just wanted to run this as a little exercise, I decided to go for a in memory implementation and not run it as a service on my machine. In hindsight this was probably a mistake and the tools and web interface would have helped me visualise my data graph quicker in the beginning.
As you can only have 1 writable instance of the in memory implementation, I made a little double lock singleton factory to create and clear the DB.
Then using Crawler4j created a graph of all the URLs starting with my blog, their relationships to other URLs and all the words and indexes of the words that those URLs contain.
After the data was collected, I could query it and perform the functions of a search engine. For this I decided to use java futures as it was another thing I had only read about and not yet implemented. In my day to day working environment we use Weblogic / CommonJ work managers within the application server to perform the same task.
I then went about creating a task for each of the following counting the word frequency, document location, Page Rank and neural network (with fake input / training data) to rank the pages returned based on the search criteria. All the code is in my public github blog repo.
Disclaimer: The Neural Network task, either didn't have enough data to be affective, or I implemented the data normalisation incorrectly, so it is currently not very useful, I'll return to it once I have completed the journey through the while PCI book.
The one task worth sharing was the Page Rank one, I quickly read some of the theory for it, decided I am not that clever and went searching for a library that had it implemented. I discovered Graphstream a wonderful opensource project that does a WHOLE lot more than just PageRank, check out their video.
From that it was then simple to implement my PageRank task of this exercise.
In between all of this I found a great implementation of sorting a map by values on Stackoverflow.
The Maven dependencies used to implement all of this
Now to chapter 5 on PCI... Optimisation.
Wednesday, July 3, 2013
Mini Search Engine - Just the basics, using Neo4j, Crawler4j, Graphstream and Encog
Labels: Encog, Graph Theory, Neo4J
Subscribe to: Post Comments (Atom)
I have recently been slacking on content on my blog, between long stressful hours at work and to the wonderful toy that is an iPhone, I have...
I make no claim to be a "computer scientist" or a software "engineer", those titles alone can spark some debate, I regar...
I saw an article (well more of a rant) the other day, by Rob Williams Brain Drain in enterprise Dev . I have to say, I do agree with some o...
This series of posts will be about me getting to grips with JBoss Drools . The reasoning behind it is: SAP bought out my company's curre...
I recently finished 97 Things every programmer should know . Well to be completely honest I did skim over a couple of the 97, but all and al...
Cool use of open src projects!ReplyDelete
Thanks, :) both the blessing and the curse of Java... so many amazing open source projects out there...Delete
Great suggestion. Very good opinion.ReplyDelete
Thanks for sharing this useful info.. voice search agencyReplyDelete
Aivivu đại lý vé máy bay, tham khảoReplyDelete
vé máy bay đi Mỹ
chuyến bay từ mỹ về việt nam
chuyến bay từ đức về hà nội hôm nay
khi nào có chuyến bay từ nga về việt nam
lịch bay từ anh về việt nam hôm nay
bay từ pháp về việt nam mấy tiếng
vé máy bay chuyên gia nước ngoài sang Việt Nam