What is your projected increase in data volume and user volume?
How scalable is your current search solution?
Rather than, Unlike:
Autonomy or Endeca.
Fast is the most linearly scalable search solution Scales in 3 different dimensions.
Volume of data – Append to the matrix, new servers to partition the index and increase the volume capacity as content grows.
Queries per second – Append to the matrix, new servers to house additional query – response services as the number of users grows.
Freshness - Append to the matrix, new servers to house additional document processing services to increase the throughput capacity.
Scalability
Business objectives, Business Challenges:
Reduce the number of servers.
Reduced total cost of ownership.
I CAN GIVE YOU AN EXAMPLE:
Associated Press were a former Autonomy customer who transitioned to FAST. When wanting to increase their query processing rate due to a growth in traffic they were advised by Autonomy to“keep adding servers”. For 5 million documents they already had 125 servers giving only 4 queries per second. FAST reduced the number of servers from 125 servers to 25. Not to give them the measly 4 QPS but 140 QPS, for complex queries on 5 million documents .
This was an 80% saving in hardware costs.
Similarly, at Autotrader.com they had previously employed a DB solution. FAST reduced the number of required servers to20 from 32. The complexity reduction meant auto trader could reduce the number of full time equivalents from 11 to one part time employee. Query processing plummeted from several seconds to a sub second response.
To summarise, this was a 47% saving in hard ware costs and a 95% saving in employee costs.
How are employees made aware of the availability of new information?
How do we assist employees in finding that useful document that they cannot fully remember the name of?
How do we facilitate the search process reducing the path to results?
FAST can suggest relevant content to employees to enable them ask better questions Auto Suggests
Rather than, Unlike:
Allow users to not only recover, but also discover what information is available. Reduce the time spent searching for information. More time spent on high-value tasks and actioning that information.
I CAN GIVE YOU AN EXAMPLE:
Cisco have a broad array of products and services. In avoid overwhelming the user they need to make their product information as consumable as possible. Cisco, powered by FAST uses auto suggest to direct users to content. This means they can find the product faster without being bogged down by information.
How can your employees currently Identify & Locate subject matter experts?
How easy is it for experts to share knowledge and insights across the organisation?
Rather than, Unlike:
Direct employees not only to documents that may contain the information but also experts they can connect to on a topic..
Business objectives, Business Challenges:
More knowledgable employees can make better decisions. Equals, greater profit.
This increases collaboration. No one is as smart as everyone.
I CAN GIVE YOU AN EXAMPLE:
We have a number of Law firms as clients. Previously, whene they dispatched a search for “corporate tax uk” they were presented with thousands of documents they would need to scour to find expertise.
Now we can expose the most relevant experts associated with a particular topic. This means we can distil the knowledge of 5 hours skimming the most relevant documents from first principles to a 5 minute phone conversation.
This means more time is spent on higher value tasks. This creates more revenue generation opportunities.
Bolster: - Respond with hard facts even if not relevant to the question. Use terms to make opponent appear aggressive The facts are hard and fast, since 1999 1000 Palestine nationals have died as a direct result of Israeli aggression.
- Personalise the response using emotion. Ask if it would be acceptable for them? How Would you feel if your daughter, mother and child were one day innocently murdered without a chance for you to say good bye. Would you accept this?
- Get them to agree on something adjacentto your argument. You do agree that the killing of young, innocent children is wrong .. DON'T YOU? (If answer no - end of argument) (if answer yes - use it to bring in adjacent argument)
Then why should the Israelis be allowed to murder young innocent children. Demerit: - Dismiss the opponents claims by showing they are not a member of the most authorative group on the subject by asking them blankly.
Are you a doctor? Well you are not exactly the most credible authority on the subject then right? (if respond - "well I am") Oh, so you are a doctor then?
- Dismiss the opponents claims by showing they are against something the audience supports. Ask them if that is what they really want to put them in defensive mode.
Your proposal will damage thousands of trees in forests accross the globe. Is that what you want?
So you don't believe collaboration is a good thing??
Let me know if you do not believe we are worth the £5 per week cost of additional training??
Winner: - Gets crowd on side - Gets the opponent to get angry
FAST can provide a proprietary active – semi-active fault tolerance within a single cluster with respect to search. With a 2 row architecture.
In this scenario, we will have two nodes – A & B – node A will host all services while node B will hold a search service, indexer service and Query/Results service. Node B can also hold additional document processing services to balance the load and increase performance.
Because both nodes hold search, indexer and Query/Results services, search is available on both nodes. This is managed by a built-in software load balancer.
This model allows us provide active – active search capabilities on both nodes in case of node failure. That is, there is search fail over. This is what is referred to as, a 2 row architecture.
A 2 column architecture would split the content across the nodes with 50% in each. This would be beneficial if there were very high volumes of data but would not provide redundancy.
With respect to content and indexing.
Node A gathers content, processes this content and uses its own indexer services to build the index.
Content dispatchers write the post processed content called FIXML (FAST Index XML) to the Node A indexer services from which the Node A Index is generated. Concurrently, this FIXML is dispatched to node B. Here node B holds the FIXML but does not yet create its own replica of the index unless the master fails.
This process is continuous to ensure both hosts' FIXML are kept in sync. If Node A fails, we can then generate the index from the Node B's FIXML and vice versa.
This process can take anywhere from minutes to several hours depending on the index size. During re-generation of Node A's index, we can still continue to serve searches from the Index Node B.
N.B. The crawler is a single point of failure. If the node containing the crawler fails, we will not be able to add new content to the index until it is brought back up.
The Indexing Subsystem.
The first indexing dispatcher to register with the Name Service becomes active. The Name Service guarantees that only one indexing dispatcher succeeds. The backup dispatchers monitor the active one, stepping in to take over if it dies. Periodically, the backup indexers connect to the master and ping it.
Incoming operations are rewritten to basic I/O operations
•e.g., invalidate file X @ position Y
•e.g., blacklist doc X in index Y_Z
Index and Search nodes are arranged in a matrix. Each column holds a subset of the content to distribute the load. This allows scale for volume and indexing performance.
Every row holds a replica of the full content of the index which enables an increased number of queries per second. Rows able to replay operations internally to re-establish synchronization after downtime. Multiple indexing rows add indexer resilience. Rows are in sync both with respect to content and indices.
The column master is elected at run-time. If the master fails, a new master is elected.
The column master synchronizes content operations and indexing to all of its backups.
During indexing of new content there will be 2 indices,
1 that houses the active index against which the search service searches. And.
1 that houses the incremenatal index that is being built from the newly added content. This is added to active index in batches.
This needs to be as large as the active index to allow for scenarios when we need to reset the entire index.
The active index will be divided into 3 partitions - 0, 1 and 2. These vary in ascending size from 25% to 50% to 100%. As content is added, the indexer service will send it to the smallest of the partitions, partition 0. When this reaches maximum capacity the index dispatcher is copied to the next largest partition 1. And so on.
The advantage of this, given that all new content is send to partition 0, is that we can now quickly re-build the smaller index partition.
The column master ensures that column contents are always in sync.
A new index is not activated until the index is ready on all rows. That is, the new index is not activated until all columns have the same content.
The master indexer is the only one to receive operations and initiate indexing, synchronizing.
Currently, all search controllers connect to the master indexer for guidance. Only the master indexer builds indices. Backups only store the FIXML. If the master fails the crown is passed to a backup, which assumes the master role.
A failover will require the index to be rebuilt from FIXML.This may take several minutes to a couple of hours depending on the volume of content. During this time indexing of new content is not possible. However, search is uninterrupted.
Processing subsystem.
Multiple processor servers provide resilience and throughput.
Multiple content distributors provide for resilience and throughput.
