What is a disruptive innovation?

In the history of literally every industry, the original technologies that shifted the industry from a cottage to more modern, were very complicated an expensive.
For example,
Telegraphs were very expensive and complicated. They had to be run by an operator who you paid to run it.

Mainframe computers were extremely expensive and complicated and had to be run by PhDs.

Then a new technology comes into the fray that is so inexpensive and so simplified that it empowers a new group of user that didn't have the money or the skills to use the incumbent product to use this disruptive offering.

For example,
Toyota versus GM.
Honda 50 motorcycles.
Skype versus established telcos.
IKEA's furniture model.
Portable radios for teens in the 60s.


This is then delivered using a cost effective disruptive business model.

Clustering, Failover and Raid.

Clustering.

In some ways, a server cluster is a little like redundant array of independent disks (RAID). With RAID, multiple physical drives work together to contain a file system. In a RAID 5 array, for example, data is striped across multiple drives, with one drive serving as a parity drive. If a drive in the array fails, you can replace that failed drive and the array rebuilds itself. If the array contains a hot spare, the array can add the spare to the array and rebuild itself without any intervention. The result is that the logical volume and data remain available.

In a server cluster, an active server can fail or be taken offline without affecting the service being provided by the cluster. In our SQL Server example, you might take the active server in the cluster offline for maintenance for several hours, but because the other server in the cluster remains online and takes over the task of serving SQL requests, customers and/or users never know the server is offline. So a cluster provides fault tolerance by allowing other servers in the cluster to take over the workload for a failed server.

In state ful clustering, the cluster maintains the user and application state during a failover, with the user and application state failing over to the other server. This means that users who access an Exchange server cluster will not lose access to their mailboxes or other Exchange features if their active server in the cluster goes down, even if they have an open connection to the server when the failure occurs.

Failover.

Failover is the ability of the cluster to move application processing from one server in the cluster to another when a hardware or application failure occurs. For example, if one of the servers in our fictitious SQL Server cluster fails, the transactions being handled by the failed server can migrate to a healthy server in the cluster. When a server comes back online in the cluster, the application can fail back to the original server.

RAID

RAID (Redundant Array of Independent Disks) is a technology that employs the simultaneous use of two or more hard disk drives to achieve greater levels of performance, reliability, and larger data volume sizes.

RAID combines two or more physical hard disks into a single logical unit by using either special hardware or software.

There are three key concepts in RAID: mirroring, the copying of data to more than one disk; striping, the splitting of data across more than one disk; and error correction, where redundant data is stored to allow problems to be detected and possibly fixed (known as fault tolerance).

When several physical disks are set up to use RAID technology, they are said to be in a RAID array. This array distributes data across several disks, but the array is seen by the computer user and operating system as one single disk.

Redundancy is a way that extra data is written across the array, which are organized so that the failure of one (sometimes more) disks in the array will not result in loss of data. A failed disk may be replaced by a new one, and the data on it reconstructed from the remaining data and the extra data. A redundant array allows less data to be stored. For instance, a 2-disk RAID 1 array loses half of the total capacity that would have otherwise been available using both disks independently, and a RAID 5 array with several disks loses the capacity of one disk. Other RAID level arrays are arranged so that they are faster to write to and read from than a single disk.

There are various combinations of these approaches giving different trade-offs of protection against data loss, capacity, and speed. RAID levels 0, 1, and 5 are the most commonly found, and cover most requirements.

• RAID 0 (striped disks) distributes data across several disks in a way that gives improved speed and full capacity, but all data on all disks will be lost if any one disk fails.

• RAID 1 (mirrored settings/disks) could be described as a real-time backup solution. Two (or more) disks each store exactly the same data, at the same time, and at all times. Data is not lost as long as one disk survives. Total capacity of the array is simply the capacity of one disk. At any given instant, each disk in the array is simply identical to every other disk in the array.

• RAID 5 (striped disks with parity) combines three or more disks in a way that protects data against loss of any one disk; the storage capacity of the array is reduced by one disk.

• RAID 6 (striped disks with dual parity) (less common) can recover from the loss of two disks.

• RAID 10 (or 1+0) uses both striping and mirroring. "01" or "0+1" is sometimes distinguished from "10" or "1+0": a striped set of mirrored subsets and a mirrored set of striped subsets are both valid, but distinct, configurations.

The configuration affects reliability and performance in different ways. The problem with using more disks is that it is more likely that one will go wrong, but by using error checking the total system can be made more reliable by being able to survive and repair the failure.

Basic mirroring can speed up reading data as a system can read different data from both the disks, but it may be slow for writing if the configuration requires that both disks must confirm that the data is correctly written. Striping is often used for performance, where it allows sequences of data to be read from multiple disks at the same time.

Indexer Fault Tolerance

Indexer Fault Tolerance .

At the end of the document processing pipeline the indexer dispatcher needs to determine which of the columns to dispatch the new content to. The location for the new document is determined by calculating:

Hash of the docid, modulo the number of columns.

This ensures that all documents have a unique location in the index.

The Name services, determines which one of the equivalent indexers is automatically elected to be the Master indexer. The other equivalent indexers are elected as backup indexers.

In a multi node installation we will have a single master Indexer and several equivalent BACK-UP indexers.

After document processing, the indexing dispatcher will dispatch the processed documents (FIXML) to all equivalent indexers on each of the rows.

Only the master indexer will actually create an index for the given document, the other indexers will receive the FIXML but not generate an index until they are required to. Such as, when the master indexer node fails.

If the master indexer fails, search will continue being served by the search node which has a full copy of the index.

Now the name service will designated a backup indexer to be master, and a new index will be generated from its FIXML.

When the original master comes back up again, this will be set up as a backup indexer.

The catch:

When indexer fault tolerance is set up a small performance degradation can be experienced because of the additional effort to copy and secure the processed documents on the backup indexer. To reduce the effect of this, it is important that the traffic between the nodes is sent on a dedicated fast (1Gbit) network.

When a backup indexer takes over, it needs to perform a reset index operation that on large indexes may take several hours to complete before it is ready to index something new.

Search will however be available throughout.

Deployment Considerations for FAST ESP

Balancing hardware COST vs. Down time Cost.

Downtime costs money, but so does redundancy! We need to weigh up the costs and tradeoffs. FAST has a built in fault tolerance mechanism called FIX ML.

Solution:

The FIXML from which index is generated could reside on a high availability SAN. Re-generation of the index from this could take in the order of minutes to hours.

Redundancy.

How is Fault Tolerance handled?
Answering.
Fault tolerant can be made with reference to 5 main subsystems.

- Administration sub-system

- Processing sub-system, including the content distributor + Document processing

- Search Subsystem, including the Query & Results Processing service + the Search service.

-Indexing Subsystem, including the index dispatcher + the indexer

- Connector Subsystem

We can scale independently each of these modules.

There are 3 fault tolerance models.

Fail Safe (full functionality if fail). For example, redundancy with multiple clusters. Careerbuilder have two identical installations, with identical crawler setups, identical document processing. One is located in a data centre on the east coast. The other in a data centre on the west coast.

Fail Soft (limited functionality if fail). For example, search fault tolerance. A 2 row architecture.

Fail Stop (No functionality if fail). For example, a single node install.

- Each service can be made safe independently. We need to balance system cost with fault tolerance requirements.
- Built-in software load balancing functionality is provided by the search API. This conducts a round robin to the QR server. This is suitable for smaller installs.

It is advisable to use an external load balancer for larger installations.

Different parts of the system can have varying degrees of fault tolerance.

Simple mirroring of the search nodes will provide high availability to the level required for most applications. This is what we refer to as adding a new row.

FAST is made up of 5 key modules or subsystems:

The search service can be configured to be Fail Safe while the content submission service can be configured to be Fail Soft.

In summary, the extra expense of adding redundancy should be calculated against the consequences of downtime and likelihood of severe failures

Follow up.
- What is the cost to the business of downtime?
- What are the likelihood of severe failures?

- How long can employees tolerate waiting for new content to be searchable? When does it become unacceptable?
- Indexing, Document processing, search can run on separate nodes? Which of these would require fault-tolerance?

What recovery processes are in built to allow recovery of the system in the event of data

loss?

FAST ESP can automatically back-up content using its native FASTXML format. In the case where there was only one row for some reason. We would advise backing up the FIXML on a SAN or tape. If a column were to go down we could re-build the index from the backed up FIXML without the requirement to re-feed and re-process the content. The FI XML can provide a system roll-back to recover data.

How do we replace a failed node or add increased capacity?

If a node goes down we can easily replace that node in isolation. FAST is a distributed system based on the Grid computing model. We can invest where it hurts.

If we need faster ingestion – we simply append a new node & increase the Doc Processors on that node.

If we need a Higher QPS – we simply append a new node & add several Query Processers to that node.

Assuming that hardware is identical among all nodes, the scaling of query rate is near-linear with respect to the number of rows since there are no interactions or co-dependencies between them.

Increasing Data volume we can add additional columns and each column will handle a partition of the entire index.

For example, 3 columns have 1/3 of the index each. The content distributor distributes the ingested documents evenly across the nodes.

Fast ESP footprint.

How does the index reflect the size of the original content? What is the Ratio? What is the installation footprint size?

Answer.

Size of the index depends on at least 50 factors including

- Document types.

- Number of navigators required.

- Number of sort able fields.

- Linguistic expansion.

- Number of queries per second.

- Rate of ingestion.

Index size can be anywhere between 20 – 200% of original content.

An example case of when it could be 200% of original size, is for a DB of rows, let’s say 2 K each.

Performing document side linguistic expansion - Adding lemmatization, entity extraction, linguistic processing. That 2 K can now become 10 K.

An example case of when it could be 20% of original size is for Word, PDF, Excel, PPT files that have massive amounts of formatting data and images. The formatting noise and images can be stripped out and the text compressed.

Response.

It may be larger but is the objective to reduce the storage size or to reduce the cost to perform more tasks in a shorter time?

We typically have on average a 30-50% smaller footprint than our competitors including DBs. How do we know? We replace a lot of them - Endeca at Bestbuy.com.

Follow Up.
- How many documents will actually be indexed?
- What types of documents? What is the make-up?
- What is the average size of each document?
- What is the growth rate of the collection?
- How many navigators are required? sortable? Integer, double, string?

Performance.
How many queries can be handled on a single node? How many documents can be handled on a single node?
ANSWERING:
On a single node FAST ESP can handle between 100 and 300 queries per second & index 3 to 5 giga bytes per hour. That is 600,000 to 1 million 500 KB word docs.

We have had instances of up to 15 million documents on a single box.
At Thompson Financial they are using our next gen product Mars to achieve roughly a 200ms indexing latency.

Based on my personal hands on experiences, I personally worked on a POC for Play. Here we were able to support up to 200 QPS for 7 million records on a single box.

FIXML- XML that follows an internal FAST xml format. It is the flat form from which the binary index is created.

fail-soft system - When system components fail, a fail-soft system continues to operate, but with reduced functionality. Such systems are also often said to provide “graceful degradation”.

fail-stop system - will not provide any functionality if system components fail. It may return false results – a situation often referred to as a “Byzantine failure”.

SAN or a NAS - “storage area network” and “network attached storage”. The servers used are remote high-performance drives shared across multiple machines, and often connected with a fibre channel.

RAID “redundant array of independent disks” – a configuration of multiple drives used to provide fault tolerance (via mirroring or parity checking) or higher performance (via striping). Frequently used to increase search- engine performance and provide a certain level of redundancy.

MTBF - It is the mean time between failures - the total elapsed time subtracted by downtime divided by the number of failures of the component.

Grid Computing The creation of a "virtual supercomputer" composed of a network of loosely-coupled computers, acting in concert to perform very large tasks.

The Advantage. Each node can be purchased as commodity hardware at lower cost than a supercomputer. Economies of producing commodity hardware, compared to the lower efficiency of designing and constructing a small number of custom supercomputers.

What do I need to know about Enterprise 2.0 and why do I need to adopt?t

Enterprise social software, also known as Enterprise 2.0, is a term describing the emerging use of social software and Web 2.0 technologies like blogs and wikis inside the firewall.

In contrast to traditional enterprise software, which imposes structure prior to use, this generation of software tends to encourage quick and easy usage firstly, and allowing structure to simply emerge.

Web 2.0 like SaaS is changing the way software is built, consumed and delivered compared with traditional software development practices.

Employee Empowerment:
Without any planning documentation, without any scoping and without any project manager - changes can quickly and easily be made by any individual in the organisation.

Moving from a Top-down paradigm where everything is structured and controlled to a bottom up model where grass roots level users are empowered to make improvements.

Communication Channels fall into 2 categories:

- Channels - person-to-person OR person-to-group - email, instant messenger - stored in a tunnel and available only to those at either end of the tunnel.

Channels create a private conduit between the sender and receiver. Other parties don't know that the email was sent, and can't consult its contents.

- Platforms - group-to-ALL OR person-to-ALL - Wikis, Blogs, Intranet sites - available to everyone and can be augmented by everyone. Information is made available to all across the organisation and is not limited to the select few that have previously communicated about it.

Wikis, del.icio.us, Flickr, Myspace, Facebook, and YouTube, on the other hand, are all platform technologies. They accumulate content over time and make it visible and accessible to all community members.

Specific Web 2.0 tools that have been adapted for enterprise use include hypertext and unstructured search tools, wikis, blogs, social bookmarking, tagging, folksonomies, RSS, social networking tools and mashups for visualization.

Problem with conventional KM software:
As McAfee states.
"Groupware actually imposed a surprising amount of structure on people's interactions, and that because Enterprise 2.0 technologies let structure emerge, rather than imposing it, they would be more popular.

Consider how high this sets the bar. Email is freeform, multimedia (especially with attachments), WYSIWYG, easy to learn and use, platform independent, social, and friendly to mouse-clickers and keyboard-shortcutters alike."

We need more 'Comfort Apps':
Email like search is what Dion Hinchcliffe calls a 'comfort app'. It is easy to use, easy to learn and universally used. We need to create such applications to proliferate Enterprise 2.0.

There exists a sharp difference between the look and feel of most corporate technologies, and most Web 2.0 ones. My favorite Web 2.0 sites are elegant, uncluttered, and bright; they have a jewel-like quality to them. I can't really say the same about most of the corporate systems I've seen and used.

Why is it difficult to get consumers and employees to change their behaviour?

As Harvard's Andrew McAfee expalins,many products fail to catch on with their intended audiences despite the clear advantages they offer.Here is a summary of his explanations:

Misconception: Consumers are not highly rational evaluators of the old vs. the new products, lining up pros and cons of each in mental tables and then selecting the winner.

Actuality [RAS]:

• We make Relative Evaluations, not Absolute ones. When I'm at a poker table deciding whether to call a bet, I don't think of what my total net worth will be if I win the hand vs. if I lose it. Instead, I think in relative terms -- whether I'll be 'up' or 'down.'

• Our reference point is the Status Quo. Comaprisons are made against where I am at that point in time. "If I win this hand I'll be up $40; if I lose it I'll be down $10 compared to my current bankroll." It's only at the end of the night that my horizon broadens enough to see if I'm up or down for the whole game.

• We are Loss Averse. A $50 loss looms larger than a $50 gainnot much affected by how much wealth one already has. Ample research has demonstrated that people find that the pain of a prospective loss of $X is about 2 to 3 times greater than a gain of $X is pleasurable.

When combined, these three lead to what the behavioral economist Richard Thaler has called the "endowment effect:"

We value items in our possession more than prospective items that could be in our possession, especially if the prospective item is a proposed substitute. One in the hand is worth two in the bush essentially.

Consumer will underweight the prospective benefits of a replacement technology by about 3X and overweight by 3X everything they're being asked to give up.

We mentally compare having the prospective item versus giving up what we already have (our 'endowment'),

but because we're loss averse giving up what we already have (our reference point) looms large.

Users are subconciously influenced by the following 3 challenges.

Challengs [GIT]:

- Timing: adopters have to give up their endowment immediately and only get (potential) benefits sometime in the future.

- Benefits are not Guaranteed: the new product might not work as promised.

- Benefits are usually Intangible: making them difficult to enumerate and compare.

Developers are blinded by their new reference point:

Gourville also highlights that the people developing new products are very dissimilar from the products' prospective consumers. You don't go work for TiVo (to use his example) if you don't 'get' the potential of digital video recorders and think they're a really good idea. And after working for the company for a while, having TiVo becomes part of your endowment; you think of things in comparison to TiVo, instead of in comparison to a VCR. Both of these factors make it harder for developers to see things as their target customers do.

Because of all of the above, Gourville talks about the '9X problem' -- "a mismatch of 9 to 1 between what innovators think consumers want and what consumers actually want."¹ The 9X problem goes a long way to explaining the tech industry folk wisdom that to spread like wildfire a new product has to offer a tenfold improvement over what's currently out there.²

Are these tools new products 9 times better than the status quo?

Solution:
There are, it seems, two broad strategies.

Increase the perceived benefits of their technologies -

lower their perceived loss and drawbacks

Demos and training are part of the former strategy, but they feel like weak measures.

Stronger ones are a clear explanation of what the technology does, network effects, peer pressure, word of mouth, incentives and an extremely effective user interface and layout.

The Power of a 'Sexy' UI:
The perceived benefits of the technology features drives them there.

But it is the UI that is going to determine what they do once they get there -- whether they'll spend time exploring and learning, or leave quickly.

A great UI not only heightens the perceived benefits of a proposed collaboration technology, it also lowers the perceived costs.

An intuitive interface lets users quickly say to themselves "Oh, I understand. This isn't hard at all. In fact, it's about as easy as email."

The greatest challenge here is making technologists sufficiently user-like -- getting them to stop thinking in terms of bells and whistles and elaborate functionality, and to start thinking instead about busy users with short attention spans who need to get something done quicker and easier.

Single search box empowers all

Single search box.

Can all employees currently extract information reports from the current CMS systems and BI tools?

What happens if they need this information?

Does this create a bottle neck?

Would it be beneficial to empower these users to extract this information themselves in real time?

Rather than, Unlike:

Complex query language interface that require SQL knowledge or programming knowledge. This creates an information asymmetry, where only those with the limited skill set can access the data.

Business objectives, Business Challenges:

Expedite adoption by all business units.

Maximize use of information assets.

Minimize learning curve and training required is zero.

I CAN GIVE YOU AN EXAMPLE:

Users resist change. It is difficult to encourage a change in behaviour. Not many people enjoy investing time to learn a new process. However, search is intuitive. We simply type in key words and are returned relevant results. You don’t see many training courses for Google?

Search fosters quick and easy adoption. It reduces the training investments. It facilitates “knowledge pull” by empowering all grass roots employees.